Abstract
OBJECTIVE: Assessment at medical schools plays a crucial role by providing feedback, monitoring student promotion, and informing resident selection. Limited research has been conducted to synthesize key features of assessments, and even less is known about how these studies sought validity and reliability evidence. A scoping review was performed to explore key features of assessments and their validity and reliability evidence. METHODS: Various databases were searched ranging from 2004 to April 2021 following PRISMA guidelines. In-depth reviews of the full text were performed on all selected empirical studies during the extraction phase. RESULTS: The final analysis included 218 empirical studies. Various tools were identified, ranging from traditional multiple-choice questions to more contemporary tools incorporating technology and more contextualized workplace-based assessment. Patient care and medical knowledge were the most frequently assessed, primarily using the quantitative methodology. These studies often adopted traditional perspectives in collecting validity evidence based on relations to other variables and internal structure. Many of them used a narrow conceptualization of validity, with some failing to reference validity at all. CONCLUSIONS: There was no single assessment that could easily differentiate medical students in a standardized, meaningful way. Despite the existence of contemporary validity theories for over two decades, there remains a need for greater education regarding the pivotal role of validity in conducting assessment research. While psychiatry and other residency programs will continue to face challenges in differentiating applicants, these changes present opportunities for medical educators and schools to validate assessments that are highly contextualized to their specific educational environments.