Abstract
INTRODUCTION: Team leadership training is essential alongside with technical training for effective resuscitation management. Addressing this gap, we developed a novel simulation system leveraging Large Language Models (LLMs) to create Artificial Intelligence (AI) agents simulating team members in Advanced Cardiovascular Life Support (ACLS) scenarios. This pilot study aimed to to develop a novel LLM-based ACLS simulation training platform and evaluate its performance in simulated resuscitation scenarios on established protocols. METHOD: Using the Claude 3.5 Sonnet API, we designed a simulation system with four AI agents assigned specific roles as healthcare staff within an ACLS team. Each agent strictly followed the 2020 American Heart Association (AHA) ACLS guidelines while interacting with an ACLS certified emergency medicine specialist user. The ten patient scenario transcripts were evaluated with three blinded emergency medicine specialists whether all the recommended steps are completed. Inter-rater reliability was assessed using Kendall’s W and Krippendorff’s Alpha statistics to evaluate agreement both within raters and the model. RESULTS: AI agents consistently adhered to the AHA 2020 ACLS algorithm across scenarios, with a high inter-rater reliability (Kendall’s W > 0.75). Krippendorff’s Alpha values for agreement ranged from substantial (0.84) to almost perfect (0.99), indicating robust compliance with guidelines and effective simulation of resuscitation responses. CONCLUSION: This study highlights the potential of LLM-powered simulations as an adjunct to traditional resuscitation training. The system effectively supported team leadership training by providing consistent and guideline-compliant responses. While the results are promising, further research with larger participant samples is necessary to evaluate the long-term educational impact and scalability of such systems. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s12909-025-08539-z.