Authors’ Response to “Putting Propensity Score Matching to Good Use in Medical Education Research”

Thank you for sharing the letter¹ about our study² and the opportunity to respond to the author’s critiques. One of the most vexing problems in educational research is claims of causality and studies of optional programs, such as the Underserved Pathway (UP), which are particularly sensitive to the issue of selection bias. We appreciate that using propensity score matching greatly decreased our sample size and yet is still subject to unobserved confounders. We also appreciate the author’s comments about generalizability and agree that although the double robust analysis supports potential generalizability of the program itself, the other variables cannot be generalized. However, these variables (age, Step one score, self-identified race) are known to influence family medicine and primary care career choice.^3-5

We performed the recommended analysis for methodological improvement, and our findings are largely unchanged. Calculation of standardized mean differences in Table 1 found that physician parent was the only variable with a Cohen d of 0.44 (medium effect size). All the other variables had a Cohen d of less than 0.1. Fixed effect modeling for family medicine practice found that Step one score was still a significant predictor for our sample, but not age (unsurprising given the initial P value = 0.049). For rural family medicine practice, race remained significant, but UP graduate status did not. All other findings were unchanged in the fixed effect models. For primary care practice, sex and Step one score remained significant; for rural practice, UP graduate status and race remained significant. For rural primary care practice, UP graduate status remained the only significant finding.

As family medicine educators at a public institution, one motivation for this study was fiscal responsibility. No medical schools have unlimited resources, and institutions must make difficult decisions about how to allocate those resources to support their missions. Our driving question was a simple one: What evidence exists that indicates the Underserved Pathway supports medical student career choice aligned with institutional workforce priorities of primary care and rural practice? We appreciate the opportunity to respond to this letter, because this additional analysis suggests that our initial findings largely hold and that the UP remains an important educational program for rural practice in general and for rural primary care more specifically.

The authors thank Doug Schaad, PhD, and Matt Cunningham, PhD, at the UW School of Medicine Division of Biomedical Informatics and Medical Education for their help collating data from matriculation and graduation surveys to help us build our matched cohort.

The authors thank Eric Larson, PhD, who was the Director of the WWAMI Rural Health Research Center, for helping us identify and interpret RUCA codes, which was key to our study methods.

The team is also grateful to John Cromartie, PhD, geographer, Economic Research Service, USDA, who provided up-to-date RUCA codes and census tracts for our data set.

Authors Ayan Mohamed, MPH, and Mansi Shah, MD, were employed in the Department of Family Medicine, Medical Student Education Section, University of Washington School of Medicine, Seattle, Washington, during the course of this project.