Project number
26055
Organization
ACABI, supported by Craig M. Berge Dean's Fund
Offering
ENGR498-F2025-S2026
Project Background/Scope: The health care encounter – whether In the medical office, clinic, hospital, home or field is critical in obtaining vital information to guide and direct diagnosis, delivery and accuracy of healthcare. Studies have shown that > 70% of diagnoses stems from the physician or health worker carefully questioning and observing the patient. Sadly, patient encounters today have become shorter, with the physician hampered while performing an exam by the burden of electronic health record (EHR) data entry and use of a computer. Studies have also shown that many correct diagnoses are made by the doctor using information, such as: what and in what way the patient speaks, how the patient looks and acts, how the patient behaves, how the patient sits, how the patient walks, and other information gained by focused, attentive, one–on–one patient observation and consultation. In routine doctor-patient interactions today much of this information is not being observed or recorded and is lost!
This project will develop an AI based tool system to be used in a “Smart Patient Exam Room” or any “Exam Space” to capture information that physicians often miss AS WELL AS DEVELOP TOOLS THAT GO BEYOND – TO EXTRACT NEW INFORMATION – creating DIGITAL BIOMARKERS OF DISEASE. This project benefits from and builds upon work done by previous Sr. Design teams who built a basic system to capture sound and image; and from a dedicated room in COM-T for this purpose allocated to this project by the medical school!
Requirements: I. Hardware – re-up a kit for sound and visual capture – can benefit from prior Sr Design team – kit should be portable (for use in any space) and fixed (for use in dedicated exam room in med school), including high fidelity microphones and cameras. II. Software/Computational Tools/AI. The team will build three sets of tools:
1. Voice to text Symptom Frequency Index and related Common/Keyword/other word Index Analysis Tools.
Step 1: Develop a dictionary of diagnostic terms from the medical “review of systems” (symptoms and signs) and from short recordings of patients with specific diseases, e.g., a patient with dyspnea might say, "I have been having a hard time catching my breath recently. I can't walk around the grocery store without stopping to catch my breath..." all this forming the analytic lexicon. Step 2: Voice to Text and Word Frequency Analysis Using a speech to text system, such as Whisper, translate the recorded audio to text. Analyze all words spoken and record their frequency. Then compare the patient’s speech to the dictionary to identify all diagnostic terms/keywords. Then create a rank scale symptom and sign frequency index, including such endpoints as: #times a word was used over an entire conversation, % Diagnostic term used = #times keyword used /all words; inter-word frequency of specific terms; and other endpoints TBD.
Step 3: Speech semantic and sentiment analysis tools – Develop semantic and sentiment analysis tools to extract meaning and emotional content to develop additional verbal quantitative biomarkers. Step 4: Develop standardized report – to display data, upload to EHR or secure cloud, printout. Keywords and endpoints can be graphed and used to compare across "multiple visits" seeing how often the patient uses the diagnostic words, e.g., a patient with dyspnea reports that they are “short of breath” 15 times in their first visit. In subsequent visits they report that they are short of breath 9, 3, 1 and 0 times. This would indicate improvement. Step 5: AI Assessment of Diagnosis and Therapy - Determine suggested diagnosis and next steps (diagnosis and therapy via querying open and closed Gen AI (ChatGPT 5 and Claude vs Llama)
2. Facial Affect/Mood/Happiness Indicator - using cameras and facial recognition the team will design a system to analyze mood and happiness based on facial expressive characteristics, grimace and related visual expressive signs. AI will be utilized and machine learning to refine the diagnostic readout.
3. Motion analysis tool - The team will use Google Mediapipe to record patient motion, create a visual skeleton of motion that may be played back and develop basic quantitative gait information in terms of speed, symmetry, stability, sit to stand and related variables that may be compared serially
NOTE: For all tools the team will develop: 1. a record keeping and display system for serial trend analysis and 2. Integrate raw and processed data into means of storage and recall from an electronic health record. Team will present data on a Screen/Heads-up display for easy analysis serially by the physician
This project will develop an AI based tool system to be used in a “Smart Patient Exam Room” or any “Exam Space” to capture information that physicians often miss AS WELL AS DEVELOP TOOLS THAT GO BEYOND – TO EXTRACT NEW INFORMATION – creating DIGITAL BIOMARKERS OF DISEASE. This project benefits from and builds upon work done by previous Sr. Design teams who built a basic system to capture sound and image; and from a dedicated room in COM-T for this purpose allocated to this project by the medical school!
Requirements: I. Hardware – re-up a kit for sound and visual capture – can benefit from prior Sr Design team – kit should be portable (for use in any space) and fixed (for use in dedicated exam room in med school), including high fidelity microphones and cameras. II. Software/Computational Tools/AI. The team will build three sets of tools:
1. Voice to text Symptom Frequency Index and related Common/Keyword/other word Index Analysis Tools.
Step 1: Develop a dictionary of diagnostic terms from the medical “review of systems” (symptoms and signs) and from short recordings of patients with specific diseases, e.g., a patient with dyspnea might say, "I have been having a hard time catching my breath recently. I can't walk around the grocery store without stopping to catch my breath..." all this forming the analytic lexicon. Step 2: Voice to Text and Word Frequency Analysis Using a speech to text system, such as Whisper, translate the recorded audio to text. Analyze all words spoken and record their frequency. Then compare the patient’s speech to the dictionary to identify all diagnostic terms/keywords. Then create a rank scale symptom and sign frequency index, including such endpoints as: #times a word was used over an entire conversation, % Diagnostic term used = #times keyword used /all words; inter-word frequency of specific terms; and other endpoints TBD.
Step 3: Speech semantic and sentiment analysis tools – Develop semantic and sentiment analysis tools to extract meaning and emotional content to develop additional verbal quantitative biomarkers. Step 4: Develop standardized report – to display data, upload to EHR or secure cloud, printout. Keywords and endpoints can be graphed and used to compare across "multiple visits" seeing how often the patient uses the diagnostic words, e.g., a patient with dyspnea reports that they are “short of breath” 15 times in their first visit. In subsequent visits they report that they are short of breath 9, 3, 1 and 0 times. This would indicate improvement. Step 5: AI Assessment of Diagnosis and Therapy - Determine suggested diagnosis and next steps (diagnosis and therapy via querying open and closed Gen AI (ChatGPT 5 and Claude vs Llama)
2. Facial Affect/Mood/Happiness Indicator - using cameras and facial recognition the team will design a system to analyze mood and happiness based on facial expressive characteristics, grimace and related visual expressive signs. AI will be utilized and machine learning to refine the diagnostic readout.
3. Motion analysis tool - The team will use Google Mediapipe to record patient motion, create a visual skeleton of motion that may be played back and develop basic quantitative gait information in terms of speed, symmetry, stability, sit to stand and related variables that may be compared serially
NOTE: For all tools the team will develop: 1. a record keeping and display system for serial trend analysis and 2. Integrate raw and processed data into means of storage and recall from an electronic health record. Team will present data on a Screen/Heads-up display for easy analysis serially by the physician