Abstract
Introduction The Chat Generative Pretrained Transformer (ChatGPT) has developed rapidly and is used in many fields, including healthcare informatics. This study evaluated ChatGPT (GPT-4V)'s performance on the Healthcare Information Technologist (HCIT) certification exam in Japan, which assesses certified professionals who work with electronic health records to improve patient care. Methodology Four hundred seventy-six questions from the HCIT exam were targeted over three years. ChatGPT (GPT-4V) was tested on its ability to answer questions from an HCIT exam to determine if it could perform as well as or better than aspirants taking the exam. Moreover, its performance was evaluated for each academic category, format, presence or absence of images, and calculations. Results The mean correct answer rate for all questions was 84%. ChatGPT achieved the exam passing criteria. The correct answer rate for simple-choice (A-type) questions was higher than that for multiple-choice (X2-type) questions (P < 0.05). The success rate for questions with images was lower than for text-only questions (P < 0.01), and the success rate for questions requiring calculations was lower than for those without calculations (P < 0.05). Conclusions ChatGPT (GPT-4V) met the passing criteria for the 19th to 21st HCIT exams, suggesting that its performance is effective in passing the HCIT exam. ChatGPT may possess the minimum required knowledge, understanding, and application skills for the HCIT certification.