Abstract
Background The growing emphasis on improving patient safety over the past two decades has received more focus in the undergraduate curricula, and the appropriate assessment of patient safety competencies at graduation is crucial in competency-based medical education. However, there is no valid method for assessing patient safety competencies because current assessment methods in medical education focus less on behavior. The objective structured clinical examination (OSCE) is a method to assess clinical performance and has been implemented by medical schools in Japan for summative assessment at graduation. However, stations with sufficient validity to assess patient safety competencies have not yet been developed. Thus, this study aimed to evaluate, under a contemporary validity framework, an OSCE station for assessing patient safety competencies that students are expected to achieve at graduation from medical schools in Japan. Methods A validity argument was conducted using Messick's validity framework, which includes content, response process, relations to other variables, internal structure, and consequences. First, we applied a modified Delphi study to develop OSCE stations for assessing patient safety competencies based on the national model core curriculum at graduation. The panel survey recruited members who have expertise in clinical education and patient safety. The draft stations simulated various situations associated with patient safety. Final-year medical students then took the OSCE. We analyzed the results of the OSCE, compared the scores with those of the clinical reasoning examination, and evaluated its reliability. Results Out of 30 panelists, 22 (73.3%) fully participated in the Delphi rounds. After two Delphi rounds, we established four stations to assess patient safety competencies. They met the content dimension of the validity framework. The OSCE results showed low correlation with clinical reasoning, suggesting that patient competencies cannot be inferred from clinical reasoning. Each station had satisfactory reliability. The entire process minimized possible assessment bias. Conclusions The OSCE scenario designed through the modified Delphi study met the five criteria of Messick's validity framework. The results show that it is a valid strategy for assessing patient safety competencies at graduation.