Abstract
Emerging evidence has shown that pre-trained encoder transformer models can extract information from unstructured clinic note text but require manual annotation for supervised fine-tuning. Large, Generative Pre-trained Transformer (GPT) models may streamline this process. In this study, we explore GPTs in zero- and few-shot learning scenarios to analyze clinical health records. We prompt-engineered Llama2 13B to optimize performance in extracting seizure freedom from epilepsy clinic notes and compared it against zero-shot and fine-tuned Bio + ClinicalBERT (BERT) models. Our evaluation encompasses different prompting paradigms, including one-word answers, elaboration-based responses, prompts with date formatting instructions, and prompts with dates in context. We found promising median accuracy rates in seizure freedom classification for zero-shot GPTs: one-word-62%, elaboration-50%, prompts with formatted dates-62%, and prompts with dates in context-74%. These outperform the zero-shot BERT model (25%) but fall short of the fully fine-tuned BERT model (84%). Furthermore, in sparse contexts, such as notes from general neurologists, the best performing GPT (76%) surpasses the fine-tuned BERT model (67%) in extracting seizure freedom. This study demonstrates the potential of GPTs in extracting clinically relevant information from unstructured EHR text, offering insights into population trends in seizure management, drug effects, risk factors, and healthcare disparities. Moreover, GPTs exhibit superiority over task-specific models in contexts with the potential to include less precise descriptions of epilepsy and seizures, highlighting their versatility. Additionally, simple prompt engineering techniques enhance model accuracy, presenting a framework for leveraging EHR data with zero clinical annotation. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s41666-025-00198-5.