The Relationship Between Preclinical Grading and USMLE Scores in US Allopathic Medical Schools

Pass/fail grading in the preclinical years of medical school is associated with improved student well-being.1-3 Students at schools with pass/fail grading have lower levels of stress, emotional exhaustion, and depersonalization, and are less likely to have burnout than those at schools with a grading system with three or more tiers (eg, Honors/Pass/Fail, A/B/C/F).2 Previous studies do not demonstrate significant differences in United States Medical Licensing Examination (USMLE) Step 1 and Step 2 scores after changing from tiered grading to pass/fail grading.1,3,4 However, the studies were conducted at highly selective institutions, so the results may not be generalizable to all medical schools. Average USMLE scores also increased overall during the study periods, potentially masking the effect of pass/fail grading. Although pass/fail preclinical grading has gained popularity in recent years, many schools still use tiered grading despite the benefits of pass/fail grading on student well-being.5 This may be due to concerns about negatively impacting students’ academic achievement and competitiveness for residency. To provide more generalizable and time-controlled evidence toward answering this question, we conducted a cross-sectional study of preclinical grading systems and average USMLE scores across 96 US allopathic medical schools.

Medical schools’ preclinical grading systems were identified with the 2017 Association of American Medical Colleges Medical School Admissions Requirements (MSAR), and verified with school websites.6 Only schools with pass/fail grading for the entire preclinical curriculum were considered pass/fail, and schools that reported maintaining a numeric system for ranking purposes were considered tiered. Schools reporting grading system changes after 2015 were classified by their old grading system. Median undergraduate grade point average (GPA) and Medical College Admission Test (MCAT) scores of accepted students were available for 142 US allopathic medical schools on MSAR. Median undergraduate GPA and MCAT scores were available for 99 schools, and average Step 1 and Step 2 clinical knowledge (CK) scores were available for 96 schools on the US News Grad Compass online database from 2015.7 The Brown University Institutional Review Board determined that this project was not human subjects research, and therefore did not require approval.

Chi-square was used to compare proportion of pass/fail schools with and without USMLE data. Two-tailed two-sample t tests with unequal variances were used to compare schools by grading system, and Pearson’s correlation coefficients were used to compare MSAR and US News data. Multiple linear regression was used to adjust for entering student characteristics, and one-tailed Student’s t tests were used to test for noninferiority. Noninferiority margins (d) were calculated using standard errors of difference for Step 1 and Step 2 CK (eight and nine points, respectively) reported in the USMLE Score Interpretation Guidelines, and average class size.8 Data were analyzed using Stata/SE 14.0 (StataCorp LP, College Station, TX).

In total, 79 of 142 (55.6%) US allopathic medical schools use pass/fail grading for the entire preclinical curriculum. Of the 96 schools with USMLE data available, 56 (58.3%) use pass/fail grading, which is not significantly different (P=0.35, Table 1). US News data for matriculating students were strongly correlated with MSAR data for accepted students. Pearson’s correlation coefficients were 0.96 and 0.89 for MCAT and GPA, respectively. Unadjusted medical school characteristics by preclinical grading system are shown in Table 2. Pass/fail schools had higher median GPA (P=0.047), median MCAT (P=0.002), average Step 1 score (P=0.014), and Step 1 pass rate (P=0.038). In the linear regression model to adjust for entering student characteristics, MCAT was a strong predictor of Step 1 (P<0.001, b=1.93±0.14, R2=0.66) and Step 2 CK (P<0.001, b=1.13±0.13, R2=0.45). Undergraduate GPA was not a significant predictor of Step 1 (P=0.17) or Step 2 CK (P=0.55) when added to MCAT, and was therefore not included. After adjusting for MCAT, pass/fail grading was not associated with significant differences in average Step 1 (P=0.98) or Step 2 CK (P=0.63). The 90% confidence interval of the effect of pass/fail grading on Step 1 (-1.27, 1.24) did not cross the noninferiority margin of -1.33 (P=0.043, Figure 1a). The 90% confidence interval for Step 2 CK (-1.46, 0.81) also did not cross the noninferiority margin of -1.5 (P=0.044, Figure 1b).

This study demonstrates that pass/fail preclinical grading is noninferior to tiered grading in terms of Step 1 and Step 2 CK scores. Although previous studies did not show significantly different USMLE scores after adopting pass/fail grading, to our knowledge, our study is the first to specifically test the noninferiority of USMLE scores with pass/fail versus tiered grading. We chose to test for noninferiority rather than equivalence or superiority of USMLE scores with pass/fail because given the evidence in the published literature that pass/fail is associated with improved student well-being and decreased stress and burnout, noninferior USMLE scores should warrant serious consideration of adopting pass/fail. Establishing noninferiority or equivalence with the absence of a statistically significant difference (P>0.05) frequently leads to incorrect conclusions.9 By testing for noninferiority, combined with our inclusion of a wide range of medical schools and adjustment for MCAT scores, we can more confidently conclude that pass/fail preclinical grading does not negatively impact students’ USMLE scores.

We also found that nearly half of all US medical schools have not adopted fully pass/fail preclinical grading. With the evidence presented here, we hope administrators at these institutions consider adopting pass/fail grading. In addition to benefits to student well-being, pass/fail preclinical grading may give students flexibility to explore topics such as population health, health disparities, and health policy that will be important for the future leaders of primary care and family medicine.

Our study has several limitations. The cross-sectional study design limits our ability to conclude cause and effect. There may also be inaccuracies in our data sources, the MSAR and US News Grad Compass, although the strong correlation in GPA and MCAT scores between the sources suggests there are no gross inaccuracies. We also may have misclassified schools that changed preclinical grading systems between 2015 and 2017 and did not report this change on their website or on MSAR. Some schools that describe themselves as pass/fail on MSAR and their website may also be maintaining internal rankings for Alpha Omega Alpha. These schools were categorized as tiered when this information was available, but others may not have reported this fact publicly. We were unable to include all US medical schools because USMLE data was unavailable for some schools, and our analysis of school-level data rather than individual students limits us from making conclusions about the effects of pass/fail on individual medical students. However, our school-level analysis should still inform administrators considering changing to pass/fail because this decision is made at the school level, not the individual level. We were also unable to analyze the effect of preclinical pass/fail on USMLE Step 2 Clinical Skills (CS) pass rates, because this information was not available on US News. The choice of noninferiority margins is also subject to bias, but we minimized this risk by basing the margins on the standard errors reported by the USMLE.

More research is needed to determine whether pass/fail preclinical grading impacts other measures of academic performance such as residency placement. Despite its limitations, our study provides evidence that pass/fail preclinical grading is noninferior to tiered grading in terms of USMLE scores.