Background and Objectives: Pass/fail preclinical grading is associated with improved student well-being. Studies do not demonstrate differences in United States Medical Licensing Examination (USMLE) scores after adopting pass/fail grading, but were conducted at unrepresentative, highly selective institutions, and average scores increased overall during the study periods. We investigated the effect of preclinical grading on USMLE scores across 96 US medical schools.
Methods: Preclinical grading systems were identified with the 2017 Association of American Medical Colleges Medical School Admissions Requirements. Median undergraduate grade point average (GPA) and Medical College Admission Test (MCAT) scores and average Step 1 and Step 2 clinical knowledge (CK) scores were available for 96 of 142 US medical schools on US News Grad Compass. Multiple linear regression was used to adjust for entering student characteristics, and one-tailed Student’s t tests were used to test for noninferiority. Noninferiority margins were calculated using standard errors from the USMLE Score Interpretation Guidelines and average class size.
Results: Fifty-six (58.3%) of 96 schools use pass/fail preclinical grading. After adjusting for MCAT, pass/fail grading was not associated with significant differences in average Step 1 (P=0.98) or Step 2 CK (P=0.63). The 90% confidence interval of the effect of pass/fail grading on Step 1 (-1.27, 1.24) did not cross the noninferiority margin of -1.33 (P=0.043). The 90% confidence interval for Step 2 CK (-1.46, 0.81) also did not cross the noninferiority margin of -1.5 (P=0.044).
Conclusions: Pass/fail preclinical grading is noninferior to tiered preclinical grading in terms of USMLE scores.
Pass/fail grading in the preclinical years of medical school is associated with improved student well-being.1-3 Students at schools with pass/fail grading have lower levels of stress, emotional exhaustion, and depersonalization, and are less likely to have burnout than those at schools with a grading system with three or more tiers (eg, Honors/Pass/Fail, A/B/C/F).2 Previous studies do not demonstrate significant differences in United States Medical Licensing Examination (USMLE) Step 1 and Step 2 scores after changing from tiered grading to pass/fail grading.1,3,4 However, the studies were conducted at highly selective institutions, so the results may not be generalizable to all medical schools. Average USMLE scores also increased overall during the study periods, potentially masking the effect of pass/fail grading. Although pass/fail preclinical grading has gained popularity in recent years, many schools still use tiered grading despite the benefits of pass/fail grading on student well-being.5 This may be due to concerns about negatively impacting students’ academic achievement and competitiveness for residency. To provide more generalizable and time-controlled evidence toward answering this question, we conducted a cross-sectional study of preclinical grading systems and average USMLE scores across 96 US allopathic medical schools.
Medical schools’ preclinical grading systems were identified with the 2017 Association of American Medical Colleges Medical School Admissions Requirements (MSAR), and verified with school websites.6 Only schools with pass/fail grading for the entire preclinical curriculum were considered pass/fail, and schools that reported maintaining a numeric system for ranking purposes were considered tiered. Schools reporting grading system changes after 2015 were classified by their old grading system. Median undergraduate grade point average (GPA) and Medical College Admission Test (MCAT) scores of accepted students were available for 142 US allopathic medical schools on MSAR. Median undergraduate GPA and MCAT scores were available for 99 schools, and average Step 1 and Step 2 clinical knowledge (CK) scores were available for 96 schools on the US News Grad Compass online database from 2015.7 The Brown University Institutional Review Board determined that this project was not human subjects research, and therefore did not require approval.
Chi-square was used to compare proportion of pass/fail schools with and without USMLE data. Two-tailed two-sample t tests with unequal variances were used to compare schools by grading system, and Pearson’s correlation coefficients were used to compare MSAR and US News data. Multiple linear regression was used to adjust for entering student characteristics, and one-tailed Student’s t tests were used to test for noninferiority. Noninferiority margins (d) were calculated using standard errors of difference for Step 1 and Step 2 CK (eight and nine points, respectively) reported in the USMLE Score Interpretation Guidelines, and average class size.8 Data were analyzed using Stata/SE 14.0 (StataCorp LP, College Station, TX).
In total, 79 of 142 (55.6%) US allopathic medical schools use pass/fail grading for the entire preclinical curriculum. Of the 96 schools with USMLE data available, 56 (58.3%) use pass/fail grading, which is not significantly different (P=0.35, Table 1). US News data for matriculating students were strongly correlated with MSAR data for accepted students. Pearson’s correlation coefficients were 0.96 and 0.89 for MCAT and GPA, respectively. Unadjusted medical school characteristics by preclinical grading system are shown in Table 2. Pass/fail schools had higher median GPA (P=0.047), median MCAT (P=0.002), average Step 1 score (P=0.014), and Step 1 pass rate (P=0.038). In the linear regression model to adjust for entering student characteristics, MCAT was a strong predictor of Step 1 (P<0.001, b=1.93±0.14, R2=0.66) and Step 2 CK (P<0.001, b=1.13±0.13, R2=0.45). Undergraduate GPA was not a significant predictor of Step 1 (P=0.17) or Step 2 CK (P=0.55) when added to MCAT, and was therefore not included. After adjusting for MCAT, pass/fail grading was not associated with significant differences in average Step 1 (P=0.98) or Step 2 CK (P=0.63). The 90% confidence interval of the effect of pass/fail grading on Step 1 (-1.27, 1.24) did not cross the noninferiority margin of -1.33 (P=0.043, Figure 1a). The 90% confidence interval for Step 2 CK (-1.46, 0.81) also did not cross the noninferiority margin of -1.5 (P=0.044, Figure 1b).
This study demonstrates that pass/fail preclinical grading is noninferior to tiered grading in terms of Step 1 and Step 2 CK scores. Although previous studies did not show significantly different USMLE scores after adopting pass/fail grading, to our knowledge, our study is the first to specifically test the noninferiority of USMLE scores with pass/fail versus tiered grading. We chose to test for noninferiority rather than equivalence or superiority of USMLE scores with pass/fail because given the evidence in the published literature that pass/fail is associated with improved student well-being and decreased stress and burnout, noninferior USMLE scores should warrant serious consideration of adopting pass/fail. Establishing noninferiority or equivalence with the absence of a statistically significant difference (P>0.05) frequently leads to incorrect conclusions.9 By testing for noninferiority, combined with our inclusion of a wide range of medical schools and adjustment for MCAT scores, we can more confidently conclude that pass/fail preclinical grading does not negatively impact students’ USMLE scores.
We also found that nearly half of all US medical schools have not adopted fully pass/fail preclinical grading. With the evidence presented here, we hope administrators at these institutions consider adopting pass/fail grading. In addition to benefits to student well-being, pass/fail preclinical grading may give students flexibility to explore topics such as population health, health disparities, and health policy that will be important for the future leaders of primary care and family medicine.
Our study has several limitations. The cross-sectional study design limits our ability to conclude cause and effect. There may also be inaccuracies in our data sources, the MSAR and US News Grad Compass, although the strong correlation in GPA and MCAT scores between the sources suggests there are no gross inaccuracies. We also may have misclassified schools that changed preclinical grading systems between 2015 and 2017 and did not report this change on their website or on MSAR. Some schools that describe themselves as pass/fail on MSAR and their website may also be maintaining internal rankings for Alpha Omega Alpha. These schools were categorized as tiered when this information was available, but others may not have reported this fact publicly. We were unable to include all US medical schools because USMLE data was unavailable for some schools, and our analysis of school-level data rather than individual students limits us from making conclusions about the effects of pass/fail on individual medical students. However, our school-level analysis should still inform administrators considering changing to pass/fail because this decision is made at the school level, not the individual level. We were also unable to analyze the effect of preclinical pass/fail on USMLE Step 2 Clinical Skills (CS) pass rates, because this information was not available on US News. The choice of noninferiority margins is also subject to bias, but we minimized this risk by basing the margins on the standard errors reported by the USMLE.
More research is needed to determine whether pass/fail preclinical grading impacts other measures of academic performance such as residency placement. Despite its limitations, our study provides evidence that pass/fail preclinical grading is noninferior to tiered grading in terms of USMLE scores.
We would like to acknowledge Seungsoo Kim for his assistance in creating the figures.
Presentations: Poster presented at Rhode Island American College of Physicians Annual Scientific Meeting, May 3, 2017.
- Bloodgood RA, Short JG, Jackson JM, Martindale JR. A change to pass/fail grading in the first two years at one medical school results in improved psychological well-being. Acad Med. 2009;84(5):655-662.
- Reed DA, Shanafelt TD, Satele DW, et al. Relationship of pass/fail grading and curriculum structure with well-being among preclinical medical students: a multi-institutional study. Acad Med. 2011;86(11):1367-1373.
- Rohe DE, Barrier PA, Clark MM, Cook DA, Vickers KS, Decker PA. The benefits of pass-fail grading on stress, mood, and group cohesion in medical students. Mayo Clin Proc. 2006;81(11):1443-1448.
- McDuff SG, McDuff D, Farace JA, Kelly CJ, Savoia MC, Mandel J. Evaluating a grading change at UCSD school of medicine: pass/fail grading is associated with decreased performance on preclinical exams but unchanged performance on USMLE step 1 scores. BMC Med Educ. 2014;14(1):127.
- Association of American Medical Colleges. Number of Medical Schools Using Selected Grading Systems. https://www.aamc.org/initiatives/cir/406418/11.html. Accessed March 3, 2017.
- Association of American Medical Colleges. Medical School Admission Requirements.https://students-residents.aamc.org/applying-medical-school/applying-medical-school-process/deciding-where-apply/medical-school-admission-requirements/. Accessed March 3, 2017.
- US News and World Report. US News Grad Compass. https://www.usnews.com/usnews/store/grad_school_compass. Accessed March3, 2017.
- United States Medical Licensing Examination. USMLE Score Interpretation Guidelines. http://www.usmle.org/pdfs/transcripts/USMLE_Step_Examination_Score_Interpretation_Guidelines.pdf. Accessed February 25, 2017.
- Walker E, Nowacki AS. Understanding equivalence and noninferiority testing. J Gen Intern Med. 2011;26(2):192-196.