BRIEF REPORTS

Family Medicine Clerkship Directors’ Perspectives on USMLE Pass/Fail Scoring: A CERA Study

Ashley Saucier, MD | Shrestha Dubey | K'Mani Blyden | F. David Schneider, MD, MSPH

Fam Med. 2024;56(8):505-508.

DOI: 10.22454/FamMed.2024.806898

Return to Issue

Abstract

Introduction: Reports on the effects of changing the United States Medical Licensing Exam (USMLE) Step 1 examination scoring to pass/fail are evolving in the medical literature. This Council of Academic Family Medicine Educational Research Alliance family medicine clerkship directors’ study seeks to describe family medicine clerkship directors’ perceptions on the impact of incorporation of Step 1 pass/fail score reporting on students’ family medicine clerkship performance.

Methods: Ninety-six clerkship directors responded (56.8% response rate). After exclusion of Canadian schools, we analyzed 88 clerkship directors’ responses from US schools. We used descriptive statistics for demographics and responses to survey questions. We used ꭓ2 analysis to determine statistically significant associations between survey items.

Results: Clerkship directors did not observe changes in students’ overall clinical performance after Step 1 pass/fail scoring (60.8%). Fifty percent of clerkship directors reported changes in Step 1 timing recommendations in the past 3 years. Reasons included curriculum redesign (30.5%), COVID (4.5%), change in Step 1 to pass/fail (11.0%), and other reasons (3.7%). Forty-five percent of these clerkship directors did not observe a change in students’ clinical medical knowledge after Step 1 went to pass/fail. Eighty-four percent of these clerkship directors did not compare student performance on clerkship standardized exams before and after Step 1 score changes. We found no significant relationship between Step 1 timing and student performance.

Conclusions: This study represents an early description of family medicine clerkship directors’ perceived observations of the impact of Step 1 scoring changes on student performance. Continued investigation of the effects of USMLE Step 1 pass/fail scoring should occur.

INTRODUCTION

In 2022, the United States Medical Licensing Exam (USMLE) changed the Step 1 examination scoring to pass/fail. The literature reports several potential benefits of this Step 1 scoring change. This includes enhanced student well-being, increased preclerkship curriculum engagement, and the potential to expand education on topics not included in Step 1 content specifications. 1-3 However, decreased motivation to study, student concerns about distinguishing themselves to residency programs, and transfer of emphasis to Step 2 and core clerkship evaluations are possible consequences. 1, 3-6

Prior to the scoring change, most schools required students to pass Step 1 before entering clerkships. 7 The current literature on Step 1 timing reflects multiple perspectives. Surveyed medical students favor Step 1 completion directly after the preclerkship phase. 1 Schools that required Step 1 after clerkship completion observed decreases in shelf exam performance. 3 Cited benefits of Step 1 after completion of clerkships were decreased Step 1 failures and increased student autonomy regarding preparation. 3

The literature on the impact of Step 1 opens opportunities to explore observations on the impact of Step 1 scoring on curriculum and student performance. This descriptive study uses data from the Council of Academic Family Medicine Educational Research Alliance (CERA) clerkship directors’ study. The purpose of this study is to understand the impact of Step 1 scoring changes on family medicine clerkship student performance as perceived by clerkship directors.

METHODS

We developed survey questions for this descriptive study as part of the CERA clerkship directors’ study. 8 Final quantitative survey questions go through several iterations of review by external reviewers, a CERA survey editor, and an assigned mentor. Prior to implementation, the survey is piloted by a small group for clarity. The CERA survey is 68 questions, including standard demographic questions and specific questions from five accepted proposals. Table 1 highlights questions relevant to this study. The survey was sent using the SurveyMonkey (SurveyMonkey, Inc) platform to 169 clerkship directors who were given 30 days to respond. The survey was completed in May 2023, approximately 18 months after the change in Step 1 scoring. Ninety-six clerkship directors responded (56.8% response rate). The CERA study protocol was approved by the American Academy of Family Physicians Institutional Review Board prior to data collection. 8

Given the study focus on the USMLE, we excluded schools from Canada. We used descriptive statistics for demographics and responses to survey questions. We excluded missing responses in the descriptive analysis of each item. We used ꭓ2 analysis to determine statistically significant associations between survey items. We performed statistical analysis using SPSS Statistics, Windows Version 29.0 (IBM).

RESULTS

Responses were received from 88 clerkship directors across the United States. 8 These clerkship directors represented 58 public and 29 private medical schools. Clerkship directors reported varied clerkship structures (71.6% block only, 20.5% block and longitudinal, and 8.0% longitudinal only). The majority of clerkship directors reported clerkship length as either 4 (42.9%) or 6 (33.3%) weeks. Of the respondents, 81.8% of directors reported clerkship taking place in the third year of medical school.

Fifty-six percent of clerkship directors reported mandatory Step 1 study periods. Fifty percent of clerkship directors reported changes in Step 1 timing recommendations in the past 3 years. Reasons included curriculum redesign (30.5%), COVID (4.5%), change in Step 1 to pass/fail (11.0%), and other reasons (3.7%). Ninety percent of clerkship directors reported that Step 1 was recommended before starting clerkships. Seventy-three percent of clerkship directors reported a requirement to pass Step 1 prior to progressing to clerkships (Table 2).

Analysis of responses from the 41 clerkship directors reporting a change in Step 1 timing recommendations yielded the following results. Of the 9 clerkship directors that reported changes in Step 1 timing due to Step 1 scoring changes, 7 (77.8%) reported recommending Step 1 before clerkships and 6 (66.7%) required students to pass prior to clerkships. Nineteen (46%) of clerkship directors did not observe a change in students’ clinical medical knowledge after Step 1 went pass/fail. Of the clerkship directors, 31 (84%) did not observe a change in students’ overall clinical performance after Step 1 went pass/fail. Seven (84%) clerkship directors did not compare student performance on clerkship standardized exams before and after Step 1 score changes.

No clerkship directors noted an increase in student medical knowledge or clinical performance since Step 1 changed to pass/fail scoring. We found no significant relationship between a change in the timing of Step 1 and clerkship directors’ reported perception of students’ performance (Table 3).

DISCUSSION AND CONCLUSIONS

This study represents an early description of family medicine clerkship directors’ perspectives on their observed impact of Step 1 scoring changes on student performance. Yadav and Dekhne et al found that USMLE Step 1 scores decreased right after transition to pass/fail; however, that occurred 1 year after the COVID pandemic began. 9 Following the change to Step 1 pass/fail scoring, clerkship directors in our study did not observe a significant impact on student medical knowledge and overall clinical performance in the family medicine clerkship. We also found that clerkship directors are not looking at the effect of the Step 1 scoring change on standardized clerkship assessments. Medical student education leadership, with more authority to change school policy, should take a close look at the impact of Step 1 scoring on student standardized assessment performance.

Sampled schools overall have not made substantial changes to their recommendations for Step 1 timing. Schools that are adjusting Step 1 timing recommendations are mostly driven by the school’s own curriculum change. This finding supports deans’ prior perspectives that the change in Step 1 to pass fail would not be a predominate driver of curriculum change. 10

One limitation of this study was that observations were limited to only family medicine clerkship directors. Survey demographics did not specify allopathic or osteopathic schools; eight osteopathic schools were in the survey pool. Further research should be done before more broad generalizations can be derived. The number of responses for schools that made changes in Step 1 recommendations due to Step 1 pass/fail scoring limited the power of analysis. Clerkship directors that affirmed changes in timing of Step 1 recommendations were not asked about prior recommendations or the specific implemented changes. Lastly, schools that are considering making changes to Step 1 timing recommendations but have not had enough time to implement them are not represented in this study. Future studies are warranted to explore actual observed benefits and consequences of making Step 1 pass/fail.

Presentations

Saucier A, Dubey S, Schneider D, Blyden K. Clerkship Directors’ Observations on Potential Impact of Step 1’s Shift From Scores to Pass/Fail. Poster Presentation. Society of Teachers of Family Medicine Conference on Medical Student Education. February 2024.

Acknowledgments

The authors acknowledge Christy Ledford, PhD, Vice Chair for Research, Medical College of Georgia, Department of Family and Community Medicine, Augusta, Geogia, for her assistance with this study.

References

  1. Cangialosi PT, Chung BC, Thielhelm TP, Camarda ND, Eiger DS. Medical students’ reflections on the recent changes to the USMLE Step exams. Acad Med. 2021;96(3):343-348. doi:10.1097/ACM.0000000000003847
  2. Carmody JB, Green LM, Kiger PG, et al. Medical student attitudes toward USMLE Step 1 and health systems science—a multi-institutional survey. Teach Learn Med. 2021;33(2):139-153. doi:10.1080/10401334.2020.1825962
  3. Daniel M, Hauer KE, Chandran L, Pock A, Morrison G, Santen SA. The optimal timing of Step 1 in medical education following the transition to pass/fail: a unique perspective from post-clerkship Step 1 schools. Med Sci Educ. 2021;31(2):905-910. doi:10.1007/s40670-021-01237-7
  4. Pontell ME, Makhoul AT, Ganesh Kumar N, Drolet BC. The change of USMLE Step 1 to pass/fail: perspectives of the surgery program director. J Surg Educ. 2021;78(1):91-98. doi:10.1016/j.jsurg.2020.06.034
  5. Belovich AN, Bahner I, Bonaminio G, et al. USMLE Step-1 is going to pass/fail, now what do we do? Med Sci Educ. 2021;31(4):1,551-1,556. doi:10.1007/s40670-021-01337-4
  6. Panda N, Bahdila D, Abdullah A, Ghosh AJ, Lee SY, Feldman WB. Association between USMLE Step 1 scores and in-training examination performance: a meta-analysis. Acad Med. 2021;96(12):1,742-1,754. doi:10.1097/ACM.0000000000004227
  7. American Association of Medical Colleges. Curriculum reports: USMLE timing reports at US medical schools, time in the curriculum in which medical schools require students to take the USMLE. Accessed November 22, 2023. https://www.aamc.org/data-reports/curriculum-reports/data/usmle-timing-requirements-us-medical-schools
  8. Kost A, Moore MA, Ho T, Biggs R. Protocol for the 2023 CERA clerkship director survey. PRiMER. 2023;7:30. doi:10.22454/PRiMER.2023.238868
  9. Yadav S, Dekhne A, Harikrishnan S, et al. The pass/fail effect: a longitudinal study of United States medical licensing examination (USMLE) Step 1 performance over a decade. Cureus. 2023;15(7):e41702. doi:10.7759/cureus.41702
  10. Manstein SM, Laikhter E, Kazei DD, Comer CD, Shiah E, Lin SJ. The upcoming pass/fail USMLE Step 1 score reporting: an impact assessment from medical school deans. Plast Surg (Oakv). 2023;31(2):169-176. doi:10.1177/22925503211034838

Lead Author

Ashley Saucier, MD

Affiliations: Department of Family and Community Medicine, Medical College of Georgia, Augusta, GA

Co-Authors

Shrestha Dubey - Department of Family and Community Medicine, Medical College of Georgia, Augusta, GA

K'Mani Blyden - Department of Family and Community Medicine, Medical College of Georgia, Augusta, GA

F. David Schneider, MD, MSPH - Department of Family Medicine, The University of Texas Southwestern Medical Center, Dallas, TX

Corresponding Author

Ashley Saucier, MD

Correspondence: Department of Family and Community Medicine, Medical College of Georgia, Augusta, GA

Email: asaucier@augusta.edu

Fetching other articles...

Loading the comment form...

Submitting your comment...

There are no comments for this article.

Downloads & Info

Share

Related Content

Tags

Searching for articles...