— RESEARCH BRIEF —

Family Medicine Residents' Debt and Certification Examination Performance

Julie P. Phillips, MD, MPH | Lars E. Peterson, MD, PhD | Iris Kovar-Gough, MA, MLIS | Thomas R. O'Neill, PhD | Michael R. Peabody, PhD | Robert L. Phillips Jr, MD, MSPH

PRiMER. 2019;3:7.

Published: 2/27/2019 | DOI: 10.22454/PRiMER.2019.568241

Abstract

Introduction: Previous studies have found that medical students and internal medicine residents with high educational debt perform less well on examinations. The purpose of this study was to examine the relationship between educational debt and family medicine residents’ performance on initial in-training and board certification examinations.

Methods: Our study was a cross-sectional secondary analysis of American Board of Family Medicine (ABFM) data collected from residents (N=5,828) who registered for the Family Medicine Certification Examination (FMCE) in 2014 and 2015, representing 85.8% of graduating family medicine residents in the United States in those years. Multivariable linear and logistic regression modeling was used to examine the relationship between debt level and examination scores, and also to explore the relationship between debt level and passing the initial FMCE.

Results: After controlling for demographic variables, residents with high debt ($150,000 to $249,999) and very high debt (more than $250,000) performed significantly worse than those with no debt on the initial in-training examination (score differences of 14.2 [CI 8.6, 19.8] and 15.8 [CI 10.3, 21.4] points, respectively) and FMCE (score differences of 19.3 points [CI 13.4, 25.3] and 30.4 points [CI 24.6, 36.3], respectively). Additionally, those with debt above $250,000 had half the odds of passing their initial FMCE (OR 0.45; CI 0.27-0.75).

Conclusions: High educational debt is associated with lower examination performance among family medicine residents. This may be because residents with more debt have more stress or fewer day-to-day financial resources. However, confounding factors may also contribute to this association.

Introduction

Educational debt is a chronic stressor for medical students and residents. Students with high debt experience more depression symptoms and are more likely to fail medical licensing exams, leave medical school voluntarily, or be dismissed.1-6 Internal medicine residents with high debt (more than $200,000) have more burnout and lower in-training exam (ITE) scores.7 Emergency medicine residents have described debt as a constant stressor that creates feelings of helplessness and regret.8 Ultimately, both resident distress and limitations in medical knowledge may negatively affect patient care.9-12

The purpose of this study was to determine whether the known relationship between higher educational debt and lower academic performance also exists among family medicine (FM) residents.

Methods

We performed a secondary analysis of American Board of Family Medicine (ABFM) administrative and FM Certification Examination (FMCE) data for residents who graduated in 2014 and 2015 and took the FMCE the same year. To measure acquisition of medical knowledge during residency, we used in-training examination (ITE) scores from the first postgraduate year (PGY1), FMCE scores, and pass/fail result from the initial FMCE attempt in April of the third postgraduate year. These examinations are based on the same blueprint and equated using the same scale (range 200 to 800), allowing meaningful comparisons between scores.

Most demographic data, including international medical graduate (IMG) status, MD or DO degree,13 age, and gender, were gathered from ABFM administrative data. The FMCE registration questionnaire included race/ethnicity and educational debt items. These race and ethnicity data were used to create an underrepresented in medicine (URM) variable to indicate residents who were American Indian/Alaska native; black/African-American; native Hawaiian/Pacific Islander; or Hispanic/Latino.14 The registration questionnaire asks examinees to indicate their total educational debt at the end of medical school, using categorical response options.

As an initial step, standard descriptive statistics were calculated to characterize the study sample. We then conducted three parallel analyses, using (1) FMCE pass/fail result, (2) FMCE score, and (3) ITE score as outcomes, with debt level as the main predictor. For the pass/fail result analysis (1), χ2 tests were used to examine the relationship between debt level and passing the FMCE. For the FMCE and ITE score analyses (2 and 3), one-way analysis of variance (ANOVA) was used to examine the relationship between debt level and examination score. Post hoc Scheffe tests were used to assess which categories of debt level were associated with lower ITE and FMCE scores.

Finally, we used multivariable linear and logistic regression models to further examine the relationships between debt, ITE and FMCE examination scores, and initial FMCE pass/fail result, controlling for associations with demographic variables. Initially, null models were used to calculate intraclass correlation and assess total variance. All demographic variables were then added to each model (URM and IMG status, degree type, gender, and age). A random intercept was also added in order to account for clustering of residents within residency programs.15 In the analyses of FMCE scores and FMCE pass/fail results, we controlled for first postgraduate year ITE score.16 Variables were added to each regression model simultaneously, because our primary goal was to examine the relationship between debt and the outcomes of interest, rather than to build parsimonious models. We calculated the total variance in the null and full models and estimated the percent variance explained, a pseudo R2.

Analyses were conducted in SAS version 9.3 (SAS Institute Inc, Cary, NC). The American Academy of Family Physicians Institutional Review Board approved this study.

Results

The sample included 5,828 residents (Table 1), representing 93.5% of all residents registering for the examination (n=6,231) and 85.8% of all graduating US FM residents (n=6,793). The majority (66.0%) were graduates of US medical schools. Most (60.3%) had more than $150,000 in educational debt.

In the initial bivariate analyses (χ2 and ANOVA), there were significant differences between debt levels and each of our three outcomes (Table 2). There were multiple statistically significant differences in both in ITE and FMCE scores across debt levels. We found a nonlinear relationship between scores and debt; those with moderate debt ($75,000 to $149,999) performed best, with average scores 18.3 points above the overall average on the ITE and 17.6 points above the overall average on the FMCE. Those with very high debt (above $250,000) were the worst performers on both ITE and FMCE, scoring 11.0 and 18.9 points below the overall averages, respectively. In post hoc testing, very high educational debt (more than $250,000) was also significantly associated with failing the FMCE.

The regression analyses demonstrated a clear relationship between increasing debt and lower test performance (Table 3). Controlling for demographic variables and clustering in residency, those with high debt ($150 to $249,999) and very high debt (more than $250,000) performed significantly worse than those with no debt on the PGY-1 ITE, with differences of 14.2 points (CI 8.6, 19.8) and 15.8 points (CI 10.3, 21.4), respectively. The adverse association between debt and academic performance continued during the remainder of residency training, as reflected in lower FMCE scores. Debt of $150,000 to 249,999 was associated with a decrease of 11.7 points on the FMCE (CI 7.0, 16.3); debt above $250,000 was associated with a decrease of 20.5 points (CI 15.9, 25.2). Those with debt above $250,000 also had lower odds of passing their initial FMCE (OR 0.59; CI 0.36-0.98). The estimated R2 for the FMCE and ITE models were 0.48 and 0.15, respectively.

Discussion

This national study of graduating FM residents over 2 years found that high educational debt is associated with weakened academic performance. Debt of more than $250,000, in particular, was associated with a 20.5-point (CI 15.9, 25.2) decrease in FMCE scores, a large enough association to suggest a clinically meaningful difference. To give context, FM residents’ ITE scores increase by an average of 34 points between their second and third years, and 27 points between their final ITE and initial FMCE.16 These differences are comparable to those previously demonstrated in a study of internal medicine residents.7 FM residents with very high debt also had substantially lower odds of passing their initial FMCE compared to peers, even after controlling for ITE score.

These results suggest that high debt may create significant stress throughout medical education. Students with high debt have more financial concerns5 and depression symptoms.1 High debt continues to contribute to financial worries for residents,8,17 who often cope with limited financial resources by accumulating consumer debt.17 Residents with higher educational debt are more likely to experience burnout,7 which may impair learning.

Residents with fewer financial resources may also purchase fewer services that promote time for study and rest. For example, a resident with less discretionary income may choose to take a bus to work rather than own a car, forego paying for childcare on a postcall day, or avoid purchasing study materials.17 Residents with more debt are also more likely to moonlight.8,17,18 Over time, these decisions may mean less time and resources for study and sleep, eroding residents’ ability to maximize learning.

Our study has several limitations. We lacked data on socioeconomic status of residents’ families of origin, historical examination performance, academic scholarships, type of medical school (public or private), and length of undergraduate medical training; all of these could contribute to the relationship between high debt and lower academic performance. Familial socioeconomic status is associated with debt19-21 and could also correlate with exam performance. Although students prolong undergraduate medical education for many reasons, examination and course failures are prominent contributors.22

Conclusion

Our results indicate that high educational debt is negatively associated with FM resident examination performance. Medical educators should help residents manage financial stress and continue to advocate for reducing the costs of medical education borne by learners.

Acknowledgments

Financial Support: Dr Julie Phillips was supported by the American Board of Family Medicine Foundation.

Presentations: Results of this study have been presented at the Society of Teachers of Family Medicine Annual Spring Conference, San Diego, California, May 2017; and at the North American Primary Care Research Group Annual Meeting, Montreal, Canada, November 2017.

References

  1. Dyrbye LN, Thomas MR, Massie FS, et al. Burnout and suicidal ideation among U.S. medical students. Ann Intern Med. 2008;149(5):334-341. https://doi.org/10.7326/0003-4819-149-5-200809020-00008
  2. Andriole DA, Jeffe DB. Prematriculation variables associated with suboptimal outcomes for the 1994-1999 cohort of US medical school matriculants. JAMA. 2010;304(11):1212-1219. https://doi.org/10.1001/jama.2010.1321
  3. Dyrbye LN, Harper W, Durning SJ, et al. Patterns of distress in US medical students. Med Teach. 2011;33(10):834-839. https://doi.org/10.3109/0142159X.2010.531158
  4. Phillips JP, Wilbanks DM, Salinas DF, Doberneck DM. Educational debt in the context of career planning: A qualitative exploration of medical student perceptions. Teach Learn Med. 2016;28(3):243-251. https://doi.org/10.1080/10401334.2016.1178116
  5. Rohlfing J, Navarro R, Maniya OZ, Hughes BD, Rogalsky DK. Medical student debt and major life choices other than specialty. Med Educ Online. 2014;19(1):25603. https://doi.org/10.3402/meo.v19.25603
  6. Morra DJ, Regehr G, Ginsburg S. Anticipated debt and financial stress in medical students. Med Teach. 2008;30(3):313-315. https://doi.org/10.1080/01421590801953000
  7. West CP, Shanafelt TD, Kolars JC. Quality of life, burnout, educational debt, and medical knowledge among internal medicine residents. JAMA. 2011;306(9):952-960. https://doi.org/10.1001/jama.2011.1247
  8. Young TP, Brown MM, Reibling ET, et al. Effect of educational debt on emergency medicine residents: a qualitative study using individual interviews. Ann Emerg Med. 2016;68(4):409-418. https://doi.org/10.1016/j.annemergmed.2016.04.013
  9. Holmboe ES, Wang Y, Meehan TP, et al. Association between maintenance of certification examination scores and quality of care for medicare beneficiaries. Arch Intern Med. 2008;168(13):1396-1403. https://doi.org/10.1001/archinte.168.13.1396
  10. Shanafelt TD, Bradley KA, Wipf JE, Back AL. Burnout and self-reported patient care in an internal medicine residency program. Ann Intern Med. 2002;136(5):358-367. https://doi.org/10.7326/0003-4819-136-5-200203050-00008
  11. Fahrenkopf AM, Sectish TC, Barger LK, et al. Rates of medication errors among depressed and burnt out residents: prospective cohort study. BMJ. 2008;336(7642):488-491. https://doi.org/10.1136/bmj.39469.763218.BE
  12. Hess BJ, Weng W, Holmboe ES, Lipner RS. The association between physicians’ cognitive skills and quality of diabetes care. Acad Med. 2012;87(2):157-163. https://doi.org/10.1097/ACM.0b013e31823f3a57
  13. Peterson LE, Blackburn B, Peabody M, O’Neill TR. Family physicians’ scope of practice and American Board of Family Medicine recertification examination performance. J Am Board Fam Med. 2015;28(2):265-270. https://doi.org/10.3122/jabfm.2015.02.140202
  14. Association of American Medical Colleges. Underrepresented in Medicine Definition. https://www.aamc.org/initiatives/urm/. Accessed May 25, 2018.
  15. Mims LD, Mainous AG III, Chirina S, Carek PJ. Are specific residency program characteristics associated with the pass rate of graduates on the ABFM certification examination? Fam Med. 2014;46(5):360-368.
  16. O’Neill TR, Li Z, Peabody MR, Lybarger M, Royal K, Puffer JC. The predictive validity of the ABFM’s in-training examination. Fam Med. 2015;47(5):349-356.
  17. Collier VU, McCue JD, Markus A, Smith L. Stress in medical residency: status quo after a decade of reform? Ann Intern Med. 2002;136(5):384-390. https://doi.org/10.7326/0003-4819-136-5-200203050-00011
  18. Steiner JW, Pop RB, You J, et al. Anesthesiology residents’ medical school debt influence on moonlighting activities, work environment choice, and debt repayment programs: a nationwide survey. Anesth Analg. 2012;115(1):170-175. https://doi.org/10.1213/ANE.0b013e318248f61d
  19. Phillips JP, Weismantel DP, Gold KJ, Schwenk TL. Medical student debt and primary care specialty intentions. Fam Med. 2010;42(9):616-622.
  20. Phillips JP, Petterson SM, Bazemore AW, Phillips RL. A retrospective analysis of the relationship between medical student debt and primary care practice in the United States. Ann Fam Med. 2014;12(6):542-549. https://doi.org/10.1370/afm.1697
  21. Cooter R, Erdmann JB, Gonnella JS, Callahan CA, Hojat M, Xu G. Economic diversity in medical education: the relationship between students’ family income and academic performance, career choice, and student debt. Eval Health Prof. 2004;27(3):252-264. https://doi.org/10.1177/0163278704267041
  22. Arvidson CG, Green WD, Allen R, et al. Investing in success: student experiences in a structured, decelerated preclinical medical school curriculum. Med Educ Online. 2015;20(1):29297. https://doi.org/10.3402/meo.v20.29297

Lead Author

Julie P. Phillips, MD, MPH

Affiliations: Sparrow-MSU Family Medicine Residency Program, Michigan State University College of Human Medicine

Co-Authors

Lars E. Peterson, MD, PhD - American Board of Family Medicine, Lexington, KY

Iris Kovar-Gough, MA, MLIS - Michigan State University Libraries

Thomas R. O'Neill, PhD - American Board of Family Medicine, Lexington, KY

Michael R. Peabody, PhD - American Board of Family Medicine, Lexington, KY

Robert L. Phillips Jr, MD, MSPH - American Board of Family Medicine, Lexington, KY

Corresponding Author

Julie Phillips, MD, MPH

Correspondence: Michigan State University, Sparrow Family Medicine Residency Program, Secchia Center, 15 Michigan Street NE, Rm 381A, Grand Rapids, MI 49503. 616-234-2672. Fax: 616-234-2638

Email: Julie.Phillips@hc.msu.edu

Fetching other articles...

Mendeley

Loading metrics from Mendeley...

Loading the comment form...

Submitting your comment...

There are no comments for this article.

Downloads & Info

Share