Abstract
BACKGROUND: Gatekeeper training using the Question, Persuade, and Refer (QPR) model has become a key strategy in suicide prevention. Yet traditional QPR-based training methods are limited by their lack of scales, interactive practice and reliable assessments of skill acquisition. Generative AI (GenAI)-driven simulators can provide a novel solution for this critical gap by offering scalable and cost-effective practices that can be culturally adapted, thus democratizing access to skills training without the potential embarrassment of live roleplay. Nevertheless, the rigorous validation of AI systems as reliable evaluation tools remains an open question. This study seeks to validate the reliability of AI-based assessments in the high-stakes context of suicide prevention, thus constituting a critical step toward using GenAI for scalable skill evaluation. METHODS: Three independent experts rated 54 simulated QPR conversations to establish the empirical reliability of a GenAI simulator feedback in the context of suicide prevention training. The primary analysis compared the automated numerical scores of the AI feedback against this human benchmark (RQ1). Secondary analyses included examination of the influence of participant characteristics on ratings (RQ2) and qualitative assessment of the AI feedback for pedagogical depth and accuracy across varied performance levels (RQ3). RESULTS: The primary finding (RQ1) exhibited a moderate-to-strong positive correlation (r = 0.519-0.776) between the GenAI adherence scores and the human-rated benchmark, providing initial evidence for the tool's reliability. No significant gender-based differences were found in either GenAI or human ratings, supporting the study's aim for an unbiased tool (RQ2). Qualitative analysis demonstrated GenAI's ability to accurately identify key QPR components and deliver nuanced in-depth feedback (RQ3). CONCLUSION: This study provides critical initial evidence that GenAI can serve as a reliable feedback tool for evaluating complex crisis intervention skills. Its ability to provide consistent, scalable, and unbiased assessments opens new possibilities for accessible evidence-based training. Despite these strong foundational capabilities, the findings also highlight the need for further calibration to align GenAI's judgment more closely with expert human nuances and enable it to evolve beyond a purely performance-focused tool.