Family Medicine Residents' Debt and Certification Examination Performance

Educational debt is a chronic stressor for medical students and residents. Students with high debt experience more depression symptoms and are more likely to fail medical licensing exams, leave medical school voluntarily, or be dismissed.^1-6 Internal medicine residents with high debt (more than $200,000) have more burnout and lower in-training exam (ITE) scores.⁷ Emergency medicine residents have described debt as a constant stressor that creates feelings of helplessness and regret.⁸ Ultimately, both resident distress and limitations in medical knowledge may negatively affect patient care.^9-12

The purpose of this study was to determine whether the known relationship between higher educational debt and lower academic performance also exists among family medicine (FM) residents.

We performed a secondary analysis of American Board of Family Medicine (ABFM) administrative and FM Certification Examination (FMCE) data for residents who graduated in 2014 and 2015 and took the FMCE the same year. To measure acquisition of medical knowledge during residency, we used in-training examination (ITE) scores from the first postgraduate year (PGY1), FMCE scores, and pass/fail result from the initial FMCE attempt in April of the third postgraduate year. These examinations are based on the same blueprint and equated using the same scale (range 200 to 800), allowing meaningful comparisons between scores.

Most demographic data, including international medical graduate (IMG) status, MD or DO degree,¹³ age, and gender, were gathered from ABFM administrative data. The FMCE registration questionnaire included race/ethnicity and educational debt items. These race and ethnicity data were used to create an underrepresented in medicine (URM) variable to indicate residents who were American Indian/Alaska native; black/African-American; native Hawaiian/Pacific Islander; or Hispanic/Latino.¹⁴ The registration questionnaire asks examinees to indicate their total educational debt at the end of medical school, using categorical response options.

As an initial step, standard descriptive statistics were calculated to characterize the study sample. We then conducted three parallel analyses, using (1) FMCE pass/fail result, (2) FMCE score, and (3) ITE score as outcomes, with debt level as the main predictor. For the pass/fail result analysis (1), χ² tests were used to examine the relationship between debt level and passing the FMCE. For the FMCE and ITE score analyses (2 and 3), one-way analysis of variance (ANOVA) was used to examine the relationship between debt level and examination score. Post hoc Scheffe tests were used to assess which categories of debt level were associated with lower ITE and FMCE scores.

Finally, we used multivariable linear and logistic regression models to further examine the relationships between debt, ITE and FMCE examination scores, and initial FMCE pass/fail result, controlling for associations with demographic variables. Initially, null models were used to calculate intraclass correlation and assess total variance. All demographic variables were then added to each model (URM and IMG status, degree type, gender, and age). A random intercept was also added in order to account for clustering of residents within residency programs.¹⁵ In the analyses of FMCE scores and FMCE pass/fail results, we controlled for first postgraduate year ITE score.¹⁶ Variables were added to each regression model simultaneously, because our primary goal was to examine the relationship between debt and the outcomes of interest, rather than to build parsimonious models. We calculated the total variance in the null and full models and estimated the percent variance explained, a pseudo R².

Analyses were conducted in SAS version 9.3 (SAS Institute Inc, Cary, NC). The American Academy of Family Physicians Institutional Review Board approved this study.

The sample included 5,828 residents (Table 1), representing 93.5% of all residents registering for the examination (n=6,231) and 85.8% of all graduating US FM residents (n=6,793). The majority (66.0%) were graduates of US medical schools. Most (60.3%) had more than $150,000 in educational debt.

In the initial bivariate analyses (χ² and ANOVA), there were significant differences between debt levels and each of our three outcomes (Table 2). There were multiple statistically significant differences in both in ITE and FMCE scores across debt levels. We found a nonlinear relationship between scores and debt; those with moderate debt ($75,000 to $149,999) performed best, with average scores 18.3 points above the overall average on the ITE and 17.6 points above the overall average on the FMCE. Those with very high debt (above $250,000) were the worst performers on both ITE and FMCE, scoring 11.0 and 18.9 points below the overall averages, respectively. In post hoc testing, very high educational debt (more than $250,000) was also significantly associated with failing the FMCE.

The regression analyses demonstrated a clear relationship between increasing debt and lower test performance (Table 3). Controlling for demographic variables and clustering in residency, those with high debt ($150 to $249,999) and very high debt (more than $250,000) performed significantly worse than those with no debt on the PGY-1 ITE, with differences of 14.2 points (CI 8.6, 19.8) and 15.8 points (CI 10.3, 21.4), respectively. The adverse association between debt and academic performance continued during the remainder of residency training, as reflected in lower FMCE scores. Debt of $150,000 to 249,999 was associated with a decrease of 11.7 points on the FMCE (CI 7.0, 16.3); debt above $250,000 was associated with a decrease of 20.5 points (CI 15.9, 25.2). Those with debt above $250,000 also had lower odds of passing their initial FMCE (OR 0.59; CI 0.36-0.98). The estimated R² for the FMCE and ITE models were 0.48 and 0.15, respectively.

This national study of graduating FM residents over 2 years found that high educational debt is associated with weakened academic performance. Debt of more than $250,000, in particular, was associated with a 20.5-point (CI 15.9, 25.2) decrease in FMCE scores, a large enough association to suggest a clinically meaningful difference. To give context, FM residents’ ITE scores increase by an average of 34 points between their second and third years, and 27 points between their final ITE and initial FMCE.¹⁶ These differences are comparable to those previously demonstrated in a study of internal medicine residents.⁷ FM residents with very high debt also had substantially lower odds of passing their initial FMCE compared to peers, even after controlling for ITE score.

These results suggest that high debt may create significant stress throughout medical education. Students with high debt have more financial concerns⁵ and depression symptoms.¹ High debt continues to contribute to financial worries for residents,^8,17 who often cope with limited financial resources by accumulating consumer debt.¹⁷ Residents with higher educational debt are more likely to experience burnout,⁷ which may impair learning.

Residents with fewer financial resources may also purchase fewer services that promote time for study and rest. For example, a resident with less discretionary income may choose to take a bus to work rather than own a car, forego paying for childcare on a postcall day, or avoid purchasing study materials.¹⁷ Residents with more debt are also more likely to moonlight.^8,17,18 Over time, these decisions may mean less time and resources for study and sleep, eroding residents’ ability to maximize learning.

Our study has several limitations. We lacked data on socioeconomic status of residents’ families of origin, historical examination performance, academic scholarships, type of medical school (public or private), and length of undergraduate medical training; all of these could contribute to the relationship between high debt and lower academic performance. Familial socioeconomic status is associated with debt^19-21 and could also correlate with exam performance. Although students prolong undergraduate medical education for many reasons, examination and course failures are prominent contributors.²²

Our results indicate that high educational debt is negatively associated with FM resident examination performance. Medical educators should help residents manage financial stress and continue to advocate for reducing the costs of medical education borne by learners.