Abstract
BACKGROUND AND AIMS: The assessment of adverse events from endoscopic procedures is essential for successful interventions, ensuring accurate follow-up, adverse event management, and processing for quality control. Despite the critical need for structured documentation, the current practice often relies on free-text recordings, which poses challenges for scalable intervention analysis; however, the introduction of large language models (LLMs) offers a promising solution by enabling the automatic extraction of adverse event details from procedural reports without altering existing documentation practices. METHODS: We analyzed 672 endoscopy reports, using OpenAI's GPT-4 and Llama-2-based models to structure the data in JavaScript Object Notation for efficient analysis. We used an automated LLM pipeline to extract adverse events such as bleeding, perforation, and aspiration. The dataset was divided into a proof-of-concept set (PoC-S) with n = 171 reports, on which we explored prompt engineering to improve the performance of the models. The final analysis was run on an additional external test set of 501 reports. RESULTS: GPT-4 showed high accuracy, with a sensitivity of 97% and specificity of 92% in the PoC-S and 91% and 96%, respectively, in the test set. GPT-4 use in real-world settings is limited by privacy concerns. Conversely, Llama-2-based models, especially the Llama-2 variants fine-tuned for German language, demonstrated comparable performance (PoC-S: sensitivity of 94%; specificity of 92%, in the test set (TS): sensitivity of 89%; specificity of 93%) and offered a viable privacy-compliant alternative. The model effectiveness was further influenced by the method of prompt engineering, with experiments showing that the specificity and sensitivity could vary substantially based on the inclusion of specific prompt features, underscoring the importance of tailored prompt design. CONCLUSIONS: Applying LLMs to extract structured medical information, particularly from endoscopy reports, offers an efficient, scalable, and adaptable documentation method that captures adverse events accurately with a low error rate. It facilitates immediate quality reporting and reduces manual documentation efforts.