Abstract
Experts have described ways to improve peer review quality. Perspectives from expert reviewers are largely absent in the health professions education literature. To gather guidance from expert reviewers, to aid authors striving to publish and reviewers aiming to perform their task effectively. This study surveyed the Journal of Graduate Medical Education (JGME) 'Top Reviewers' from 2017, 2018, and 2019. 'Top Reviewers' perform four or more reviews per year, with high average ratings. Top reviewers were sent an 11-item survey in February 2020. The survey included three demographic questions and eight open-ended, free-text questions about the concepts reviewers most often target in their reviews. We calculated descriptive statistics and performed a thematic analysis of open-ended responses. Of 62 eligible top reviewers, 44 (71%) responded to the survey. Only eight (18.2%) and seven (15.9%) respondents reported having 'stock phrases' or a reviewer template used for reviewer feedback to authors, respectively. The what (research question, methods), how (presentation, writing), and why (relevance, impact) were the resulting themes summarizing how reviewers categorized and responded to common problems. For 'really good papers' reviewers found the what acceptable and focused on how and why. For 'really bad' papers, reviewers focused on big picture feedback, such as the value of the study. Top reviewers from a single health professions education journal appear to have similar approaches to conducting reviews. While most do not use stock phrases or templates, they share similar strategies to differentiate 'good' vs. 'bad' papers through the what, why, and how of a manuscript.