Abstract
Reflective narratives contain meaningful information about medical students' experiences and environments and are a rich source for qualitative analysis. However, analyzing narratives to learn about their content is expensive in terms of time and money. Recently developed artificial intelligence-based tools may help ease the analysis process. This study explored the potential of large language models (LLMs) to facilitate narrative text analysis. The narratives were written by medical students on their observations and reflections on medical encounters in which bad news was delivered. These narratives are challenging for analysis due to their semi-structured nature and medical and emotional content. We compared a former analysis process done manually versus the ability of LLMs to analyze these narratives, across different categories, including contextual (e.g., demographics) and thematic analysis. Our results showed that the LLM effectively processed data and followed formatting requirements. However, the LLM's performance varied across different categories, showing closer alignment with human analysis in straightforward categories (e.g., age), while struggling with more nuanced aspects of the narratives (e.g., assessing surprise). Ambiguities and complexities in the narratives posed significant challenges for the LLM, limiting its interpretive accuracy needed in such cases. The findings suggest that LLMs could complement human analysis, particularly in extracting explicitly mentioned contextual information, thereby enhancing the efficiency of the analysis process. The research highlights the need for further investigation into the capabilities of LLMs, pointing toward a future where the LLM and human qualitative researchers work in synergy to manage the demand for a timely and accurate narrative analysis.