Abstract
BACKGROUND: Direct observation of clinical skills is a cornerstone of competency-based education and training. Ensuring direct observation in a consistent fashion has been a significant challenge for residency programs. OBJECTIVE: The purpose of this study was to evaluate the effects of a novel evaluation system, designed to achieve ongoing direct observation of residents, examine changes in resident observation practices, and understand faculty attitudes toward direct observation and the evaluation system. METHODS: Internal medicine residents on an ambulatory block rotation participated in a new evaluation system, which replaced a single end-of-rotation summative evaluation with 9 formative evaluations based on direct observation. Faculty received training in direct observation and use of the forms, and residents were given responsibility to collect 9 observations per rotation. Faculty members contacted residents at the beginning and middle of the rotation to ensure completion of the observations. Residents and faculty also completed postrotation surveys to gauge the impact of the new system. RESULTS: A total of 507 patient encounters were directly observed, and 52 of 57 (91%) residents completed all 9 observations. Residents reported considerably more direct observation than prior to the intervention, and most reported changes to their clinical skills based on faculty feedback. Faculty reported improvements in their attitudes, increased their use of direct observation, and preferred the new system to the old one. CONCLUSIONS: A novel evaluation system replacing summative evaluations with multiple formative evaluations based on direct observation was successful in achieving high rates of observations, and improving faculty attitudes toward direct observation.