Abstract
Background/Objectives: Assessing non-technical skills (NTSs) in prehospital care is susceptible to rater subjectivity. While Artificial Intelligence (AI) can be used to score conversation transcripts, it emphasizes formal linguistic features, whereas humans integrate scene context, leading to potentially divergent evaluations. We examined the validity of NTS assessments generated by AI (ChatGPT-4o) from prehospital simulation data by comparing them with ratings from paramedic faculty. We hypothesized that AI-based ratings would provide evaluations of team NTSs that are comparable to faculty ratings and would enable us to describe the direction and magnitude of score differences between AI and faculty across the five NTS domains. Methods: Sixty-four first-year paramedic students performed 5 min prehospital scenarios. Five NTS domains were scored independently by AI and faculty using a three-level rubric (5, 3, or 1 point per domain): (i) communication and interpersonal manner, (ii) order and completeness of information gathering, (iii) detail of follow-up questioning, (iv) context-appropriate actions, and (v) time management. Score differences were analyzed with Wilcoxon signed-rank tests with Holm correction and Bayes factors (BF10). Agreement was quantified with weighted Gwet's agreement coefficient 2 (AC2). Results: Three domains-communication, context-appropriate actions, and time management-showed significant differences (p < 0.001), with strong evidence for differences (BF10 > 22); median differences favored AI. Evidence of a difference was insufficient for the other two domains. Across all domains, agreement remained below the prespecified substantial threshold (AC2 ≥ 0.60). The primary hypothesis was not supported. Conclusions: In prehospital simulations, AI-only NTS assessment is not yet an adequate substitute for human raters. Although AI evaluates linguistic aspects, its agreement with expert ratings was insufficient. Future work should evaluate hybrid approaches leveraging the strengths of both AI and human judgment.