Family Medicine Clerkship Directors’ Perspectives on USMLE Pass/Fail Scoring: A CERA Study

In 2022, the United States Medical Licensing Exam (USMLE) changed the Step 1 examination scoring to pass/fail. The literature reports several potential benefits of this Step 1 scoring change. This includes enhanced student well-being, increased preclerkship curriculum engagement, and the potential to expand education on topics not included in Step 1 content specifications. ^1-3 However, decreased motivation to study, student concerns about distinguishing themselves to residency programs, and transfer of emphasis to Step 2 and core clerkship evaluations are possible consequences. ^{1, 3-6}

Prior to the scoring change, most schools required students to pass Step 1 before entering clerkships. ⁷ The current literature on Step 1 timing reflects multiple perspectives. Surveyed medical students favor Step 1 completion directly after the preclerkship phase. ¹ Schools that required Step 1 after clerkship completion observed decreases in shelf exam performance. ³ Cited benefits of Step 1 after completion of clerkships were decreased Step 1 failures and increased student autonomy regarding preparation. ³

The literature on the impact of Step 1 opens opportunities to explore observations on the impact of Step 1 scoring on curriculum and student performance. This descriptive study uses data from the Council of Academic Family Medicine Educational Research Alliance (CERA) clerkship directors’ study. The purpose of this study is to understand the impact of Step 1 scoring changes on family medicine clerkship student performance as perceived by clerkship directors.

We developed survey questions for this descriptive study as part of the CERA clerkship directors’ study. ⁸ Final quantitative survey questions go through several iterations of review by external reviewers, a CERA survey editor, and an assigned mentor. Prior to implementation, the survey is piloted by a small group for clarity. The CERA survey is 68 questions, including standard demographic questions and specific questions from five accepted proposals. Table 1 highlights questions relevant to this study. The survey was sent using the SurveyMonkey (SurveyMonkey, Inc) platform to 169 clerkship directors who were given 30 days to respond. The survey was completed in May 2023, approximately 18 months after the change in Step 1 scoring. Ninety-six clerkship directors responded (56.8% response rate). The CERA study protocol was approved by the American Academy of Family Physicians Institutional Review Board prior to data collection. ⁸

Given the study focus on the USMLE, we excluded schools from Canada. We used descriptive statistics for demographics and responses to survey questions. We excluded missing responses in the descriptive analysis of each item. We used ꭓ² analysis to determine statistically significant associations between survey items. We performed statistical analysis using SPSS Statistics, Windows Version 29.0 (IBM).

Responses were received from 88 clerkship directors across the United States. ⁸ These clerkship directors represented 58 public and 29 private medical schools. Clerkship directors reported varied clerkship structures (71.6% block only, 20.5% block and longitudinal, and 8.0% longitudinal only). The majority of clerkship directors reported clerkship length as either 4 (42.9%) or 6 (33.3%) weeks. Of the respondents, 81.8% of directors reported clerkship taking place in the third year of medical school.

Fifty-six percent of clerkship directors reported mandatory Step 1 study periods. Fifty percent of clerkship directors reported changes in Step 1 timing recommendations in the past 3 years. Reasons included curriculum redesign (30.5%), COVID (4.5%), change in Step 1 to pass/fail (11.0%), and other reasons (3.7%). Ninety percent of clerkship directors reported that Step 1 was recommended before starting clerkships. Seventy-three percent of clerkship directors reported a requirement to pass Step 1 prior to progressing to clerkships (Table 2).

Analysis of responses from the 41 clerkship directors reporting a change in Step 1 timing recommendations yielded the following results. Of the 9 clerkship directors that reported changes in Step 1 timing due to Step 1 scoring changes, 7 (77.8%) reported recommending Step 1 before clerkships and 6 (66.7%) required students to pass prior to clerkships. Nineteen (46%) of clerkship directors did not observe a change in students’ clinical medical knowledge after Step 1 went pass/fail. Of the clerkship directors, 31 (84%) did not observe a change in students’ overall clinical performance after Step 1 went pass/fail. Seven (84%) clerkship directors did not compare student performance on clerkship standardized exams before and after Step 1 score changes.

No clerkship directors noted an increase in student medical knowledge or clinical performance since Step 1 changed to pass/fail scoring. We found no significant relationship between a change in the timing of Step 1 and clerkship directors’ reported perception of students’ performance (Table 3).

This study represents an early description of family medicine clerkship directors’ perspectives on their observed impact of Step 1 scoring changes on student performance. Yadav and Dekhne et al found that USMLE Step 1 scores decreased right after transition to pass/fail; however, that occurred 1 year after the COVID pandemic began. ⁹ Following the change to Step 1 pass/fail scoring, clerkship directors in our study did not observe a significant impact on student medical knowledge and overall clinical performance in the family medicine clerkship. We also found that clerkship directors are not looking at the effect of the Step 1 scoring change on standardized clerkship assessments. Medical student education leadership, with more authority to change school policy, should take a close look at the impact of Step 1 scoring on student standardized assessment performance.

Sampled schools overall have not made substantial changes to their recommendations for Step 1 timing. Schools that are adjusting Step 1 timing recommendations are mostly driven by the school’s own curriculum change. This finding supports deans’ prior perspectives that the change in Step 1 to pass fail would not be a predominate driver of curriculum change. ¹⁰

One limitation of this study was that observations were limited to only family medicine clerkship directors. Survey demographics did not specify allopathic or osteopathic schools; eight osteopathic schools were in the survey pool. Further research should be done before more broad generalizations can be derived. The number of responses for schools that made changes in Step 1 recommendations due to Step 1 pass/fail scoring limited the power of analysis. Clerkship directors that affirmed changes in timing of Step 1 recommendations were not asked about prior recommendations or the specific implemented changes. Lastly, schools that are considering making changes to Step 1 timing recommendations but have not had enough time to implement them are not represented in this study. Future studies are warranted to explore actual observed benefits and consequences of making Step 1 pass/fail.

Saucier A, Dubey S, Schneider D, Blyden K. Clerkship Directors’ Observations on Potential Impact of Step 1’s Shift From Scores to Pass/Fail. Poster Presentation. Society of Teachers of Family Medicine Conference on Medical Student Education. February 2024.