Abstract
BACKGROUND: With ambient listening systems increasingly adopted in healthcare, analyzing clinician-patient conversations has become essential. The Omaha System is a standardized terminology for documenting patient care, classifying health problems into four domains across 42 problems and 377 signs/symptoms. Manually identifying and mapping these problems is time-consuming and labor-intensive. This study aims to automate health problem identification from clinician-patient conversations using large language models (LLMs) with retrieval-augmented generation (RAG). METHODS: Using the Omaha System framework, we analyzed 5118 utterances from 22 clinician-patient encounters in home healthcare. RAG-enhanced LLMs detected health problems and mapped them to Omaha System terminology. We evaluated different model configurations, including embedding models, context window sizes, parameter settings (top k, top p), and prompting strategies (zero-shot, few-shot, and chain-of-thought). Three LLMs-Llama 3.1-8B-Instruct, GPT-4o-mini, and GPT-o3-mini-were compared using precision, recall, and F1-score against expert annotations. RESULTS: The optimal configuration used a 1-utterance context window, top k = 15, top p = 0.6, and few-shot learning with chain-of-thought prompting. GPT-4o-mini achieved the highest F1-score (0.90) for both problem and sign/symptom identification, followed by GPT-o3-mini (0.83/0.82), while Llama 3.1-8B-Instruct performed worst (0.73/0.72). CONCLUSIONS: Using the Omaha System, LLMs with RAG effectively automate health problem identification in clinical conversations. This approach can enhance documentation completeness, reduce documentation burden, and potentially improve patient outcomes through more comprehensive problem identification, translating into tangible improvements in clinical efficiency and care delivery. CLINICAL RELEVANCE: Automating health problem identification from clinical conversations can improve documentation accuracy, reduce burden, and ensure alignment with standardized frameworks like the Omaha System, enhancing care quality and continuity in home healthcare.