Abstract
Introduction This study explored the potential of using large language models (LLMs) to generate multiple-choice questions (MCQs) for the Japanese National Licensure Examination for Physical Therapists. Specifically, it evaluated the performance of a customized ChatGPT (OpenAI, San Francisco, CA, USA) model named "Physio Exam Generative Pre-trained Transformers (GPT)" in generating high-quality MCQs in non-English contexts. Materials and methods Based on the data extracted from the 57th and 58th Japanese National Licensure Examination for Physical Therapists, 340 MCQs, including correct answers, explanations, and associated topics, were incorporated into the knowledge base of the GPTs. The prompts and outputs were conducted in Japanese. The generated MCQs covered major topics in general (anatomy, physiology, and kinesiology) and practical questions (musculoskeletal disorders, central nervous system disorders, and internal organ disorders). The quality of the MCQs and their explanations were evaluated by two independent reviewers using a 10-point Likert scale across five criteria: clarity, relevance to clinical practice, suitability of difficulty, quality of distractors, and adequacy of rationale. Results The generated MCQs achieved 100% accuracy for both general and practical questions. The average scores across the evaluation criteria ranged from 7.0 to 9.8 for general questions and 6.7 to 9.8 for practical questions. Although some areas exhibited lower scores, the overall results were favorable. Conclusions This study demonstrates the potential of LLMs to efficiently generate high-quality MCQs, even in non-English environments such as Japanese. These findings suggest that LLMs can adapt to diverse linguistic settings, reduce educators' workload, and improve the quality of educational resources. These results lay a foundation for expanding the application of LLMs to educational settings across non-English-speaking regions.