Academic Achievement and Competency in Rural and Urban Family Medicine Residents

There have long been concerns over shortages of health care professionals in rural areas. ^{1, 2} Evidence suggests that exposure to rural practice during training, particularly during residency, is important to increasing both rural placement and retention of physicians. ^{3, 4} Among physicians, family physicians (FPs) are most likely to practice in a rural area ⁵ and, due to their broad training and the lower availability of other medical specialties, often enjoy a broader scope of practice. ^{6, 7} Rural family medicine residencies are ideally positioned to train comprehensive generalist physicians to practice in rural areas.

However, these rural residencies are more likely to be resource constrained in an academic sense, which has raised anecdotal concerns regarding the quality of training. ⁸ The Accreditation Council on Graduate Medical Education (ACGME) has had a long-standing minimum size requirement for accreditation, with at least four resident positions offered per educational level. ⁹ Smaller rural programs in a 1+2 format, where the residents spend their intern year at a larger standard program and their last 2 years at a smaller, more rural training site, have been permitted a variance from this minimum. Whether small rural programs are training their graduates as well as larger urban programs remains unknown.

Two studies of program-level data from 2008 to 2012 found that smaller programs had lower average passing rates on the American Board of Family Medicine (ABFM) initial certification examination. ^{10, 11} The median pass rate during this period was just below 90%, but the ABFM has since reported an increase in resident performance with pass rates rising into the mid 90s from 2009 to 2016. ¹² In particular, the latter study found an increase in pass rate for international medical graduates (IMGs), who are more common in rural programs. No study has explicitly investigated rural versus urban differences in ABFM initial examination performance.

Family medicine residents are assessed on common metrics throughout training. First, the ABFM offers an In-Training Examination (ITE) to all residencies. This formative test is offered in the fall of each year, is based on the same blueprint, and is scored on the same scale as the Family Medicine Certification Examination (FMCE). Second, each residency must report to the ACGME on 22 milestones across the six core physician competencies (patient care [PC], medical knowledge [MK], interpersonal and communication skills [ICS], professionalism [PROF], practice-based learning and improvement [PBLI], systems-based practice [SBP]) for each resident twice per year. Our objective was to compare resident academic performance on standardized tests of medical knowledge and milestones of residents in rural and urban family medicine residency programs.

We combined ABFM FMCE registration and score data from 2014-2018 with ITE scores, milestones, and ABFM administrative data on residents and residencies. We excluded residents without a complete set of examination and milestones data, as well as those who transferred programs, or were in combined programs or 4-year training programs. As some rural training track (RTT) programs require residents to do a portion of their PGY-1 year at a different clinic site (1+2 programs), we restricted milestones analyses to PGY2 and PGY3 to limit the potential of bias from a different group of evaluators.

Our independent variables included resident age, gender, race, ethnicity, degree type, IMG status, as well as residency size and program rurality. We obtained race and ethnicity data from the ABFM FMCE Registration Questionnaire for initial certification candidates and combined responses to create an underrepresented in medicine (URM) variable reflecting being American Indian or Alaska Native, Black or African American, Native Hawaiian or Other Pacific Islander, or Hispanic ethnicity. ¹³ Responses of “other” race were recoded to reflect URM status. For residents whose race/ethnicity could not be determined, we classified URM status as “unknown.” Site of medical education was categorized as United States/Canada or IMG. We calculated residency size from ABFM administrative data by counting the number of PGY-3 residents, categorized as less than 4, 4 to 10, and 11 or more. We defined rural residencies as programs where residents spend more than 50% of their time in rural locations (rural urban commuting area >4) using the list maintained by the RTT Collaborative. ¹⁴

During the study period, the milestones contained 22 items across six core competencies. Milestones ratings from 0 to 5, on an ordinal scale (eg, 3.0, 3.5, and 4.0), are assigned to each resident twice per year by the residency, midyear and end-year, roughly December and June. There are no academic consequences for milestone ratings, but the goal on each rating at the time of residency graduation is level 4 for all competencies. ¹⁵ We characterized whether the resident met expectations on each milestone at each evaluation period starting with a rating of 4 at the end of PGY-3 and removing 0.5 points for each of the preceding ratings because we used midyear ratings. Thus our metric for meeting expectations at each assessment period was a rating of 3.5 at midyear PGY3, 3.0 at end year PGY2, and 2.5 at midyear PGY2. To evaluate overall change, we created two summary measures of milestones. First, we determined the percentage of residents who met or exceeded every milestone during an evaluation overall and within each core competency. Second, we calculated the percentage of milestones the resident met or exceeded during an evaluation overall and within each core competency. Finally, a subset of authors involved in medical education (R.L., D.E., D.S.) reviewed the milestones and hypothesized that rural residents may have better performance than urban residents on seven specific milestones (PC-4, PC-5, SBP-3, SBP-4, PBLI-2, PROF-3, and PROF-4). ⁹ For these we calculated the percentage of residents meeting the milestone at each rating.

ITE and FMCE performance scores range from 200 to 800. The ITE is intended to be a formative evaluation of the resident’s current medical knowledge and has no passing score. During our study period the passing score for FMCE was 380, and we created a pass/fail variable for the initial attempt because that result has been used for ACGME accreditation. Those who fail may take the FMCE again. To determine eventual pass rate, we captured any examination within 3 years of residency graduation.

We conducted t tests or χ² tests for associations of variables with program type. To determine the relative meaningfulness of any differences found, we calculated a Cohen’s D or H for continuous and categorical outcomes, respectively. ¹⁶ To assess differences in milestones performance, we first plotted the two summary milestone performance measures at each rating overall and by core competency for rural and urban programs. We then plotted the seven specific milestones for which we hypothesized rural residents would perform better. Next, we conducted multilevel logistic regression controlling for potential confounders including resident demographics (including degree type and IMG status), residency size, and clustering at the residency level to assess independent associations of failing to meet any milestone at the end of PGY3 and failing to meet any of the seven specific milestones hypothesized to have differential rural/urban performance.

To assess acquisition of medical knowledge via examination performance, we graphed ITE to FMCE performance and tested for differences by residency type at each examination instance. We conducted multilevel linear regression to test for association of rural training and examination score using the same variables, including confounders, as in the milestones model. Interaction terms between program type and examination instance assessed for differential growth in scores. A multilevel logistic regression, using the same variables, including confounders, as in the previous models assessed independent associations between initial FMCE failure and residency type. An additional logistic regression model determined associations with subsequent FMCE failure and residency type. All models controlled for resident and residency characteristics. We used SAS software version 9.2 (SAS Institute, Cary, NC) for all analyses. The American Academy of Family Physicians Institutional Review Board approved this study.

We identified 16,456 residents who completed 3-year core family medicine residency training from 2016 to 2018, then excluded those who transferred or lacked exam score data or milestone data, yielding two analytic samples: 11,790 FPs with complete exam data and 11,613 with complete milestones data. Sample descriptions are shown for the exam cohort as the descriptive statistics are virtually identical between these samples. Only 550 (4.7%) of the exam cohort were in rural programs (Table 1). Rurally-trained residents were less likely to be URM (9.1% vs 17.6%), MDs (76.4% vs 80.0%), US/Canadian graduates (59.1% vs 66.7%), and more likely to be in smaller programs than urban-trained residents. Overall, 98.8% of residents passed their initial FMCE attempt with rural residents having a lower overall rate compared to urban residents (96.2% vs 98.9%). Extending our examination period to all attempts within 3 years following residency graduation, of the 11,790 residents, only 32 failed all attempts. Six of those were rural graduates. There was no significant difference in initial FMCE pass rate by program type in bivariate analysis. Effect sizes for both FMCE score and pass rate were small at 0.16 and 0.19, respectively.

There was no significant difference between rural and urban residents on PGY-1 ITE scores (Figure 1). By PGY-2 ITE, rural residents scored 11.1 points lower, a trend that continued to the initial FMCE where they scored 12.7 points lower than urban residents. Adjusting for resident and residency characteristics in a multilevel longitudinal model, rural residency was not associated with FMCE score (β=-2.1 95% CI [-8.4, 4.2]; Table 2). Additionally, interaction terms between examination instance and program type were not significant, indicating equal growth in scores between program types. In adjusted models for odds of failing the initial FMCE, rural residents did have higher odds of failing (OR=3.32, 95% CI [1.95, 5.66]). In models capturing subsequent exam attempts 3 years from residency graduation, rural residents had higher odds of failing (OR=3.23, 95% CI [1.13, 9.20], Supplemental Table 1).

There was no significant difference in the percentage of residents meeting all milestones at midyear PGY2 by training program (25.2% vs 28.0%), but by end-year, PGY-3 rural residents were less likely to meet all milestones (14.9% vs. 23.5%; Figure 2). Rural and urban residents had similar milestone ratings on specific core competencies at PGY-2 midyear except in professionalism, where rural residents had lower ratings (51.5% vs 61.7%). By end-year PGY3, fewer rural than urban residents met all milestone expectations: 8.5% overall and differing by 6.5% to 17.4% on specific core competencies. Examining percentages of milestones met at each rating, a similar pattern of no difference was seen at midyear PGY2, but there were fewer differences by end-year PGY3 (Supplemental Figure 1). The percentages met at end-year PGY-3 was lower for rural residents by 5.7% to 13.0% for milestones overall, SBP, PBLI, and PROF. For the seven specific milestones we hypothesized would vary by program type, there were no differences at midyear PGY2, but rural residents were less likely to meet four of the milestones at end-year PGY3 by 6.1% to 12.3%, compared to urban residents (Supplemental Figure 2). Effect sizes for differences in milestones ratings were small and ranged from 0.02 to 0.26. Regression analysis found consistent positive associations between failing to meet milestones expectations—overall and for seven specific milestones—and being an IMG (all eight models, OR’s 1.30 to 1.55), a DO (5 of eight models, ORs 1.14 to 1.43), and in a rural program (five of eight models, OR’s 1.31 to 1.79; Table 3).

Using a large sample of recent family medicine graduates, we found small but persistent differences in academic performance on ABFM examinations and in attainment of milestones in multiple competencies by rural program residents compared to urban residents. The practical meaning of these differences is less clear and the material impact on clinical practice and workforce policy need to be explored further.

For examination performance, while the odds of failing the initial, and subsequent, attempts are three times higher for rural trainees, the overall pass rate on the initial exam is only 2.7 percentage points lower than that of urban residents and the eventual pass rate for rural graduates is 98.9%. These results are reassuring in that only a small proportion of residents do not acquire the requisite medical knowledge to pass their board examination. Our results should be viewed in the context of improvements in pass rate among all residents with the overall pass rate for residents increasing from 91% to 98% in the years immediately preceding our data, ¹² compared to an overall pass rate for practicing physicians of 89.7% in 2018. ¹⁷

Medical knowledge, although foundational, is only one of six physician core competencies. In analysis of milestones ratings across all core competencies, we found that rural residents were largely comparable to urban residents at the beginning of PGY2. However, they were persistently less likely to meet summary measures of milestones overall and by core competency by the end of training. Differences in the trajectory of outcomes in milestones summary measures may be due to smaller program size. Rural residents may be more closely scrutinized as faculty evaluators are more likely to have first-hand knowledge of residents’ performance, may be less likely to give trainees the benefit of the doubt, and may be able to give more specific and individualized feedback and assessment than faculty in larger programs. Small rural programs may assess milestones differently than large programs based on the frequency of interaction and direct observation with each individual resident by peers and faculty. The milestones hypothesized to have higher performance in rural programs by our authors with rural faculty experience, are also consistent with higher rural faculty expectations of residents. All these potential explanations are ripe for further study to understand how variation in assessments in rural programs develops and whether such variation is due to program or resident characteristics, or their interaction.

There has been debate whether the milestones measure externally-consistent resident performance or interprogram variation and intraprogram performance. ^{18, 19} An analysis of family medicine milestone trajectories, using the same data from one of our cohorts, was done by the ACGME. In this analysis, residency level factors accounted for 35% of the variance in milestone ratings, while individual resident characteristics accounted for 22% of the variance. ²⁰ These results suggest substantial variability due to program structure, Clinical Competency Committee function, and program culture, and that these factors may influence progression more than the individual resident.

Using the 2014-2015 Milestones, researchers found little variability in the milestones ratings at each time point. ²¹ However, later studies found that as programs gained experience with the milestones, variation in ratings increased. ²² The 2020 Milestones report by the ACMGE found “straight-lining,” where residents receive the same rating at each time point, occurred in only 8.3% of PGY-1, 4.5% of PGY-2, and 5.6% of PGY-3 residents. ¹⁵ Park et al found the variability in milestones at both the resident and residency level decreased during a resident’s training. ²⁰ These findings support the use of milestones ratings later in training to assess program performance.

Although board examinations and milestones are the only measures of resident performance in wide use, they may not fully measure the skills and knowledge imparted in residency and those needed for competent practice. The milestones are relatively new and evidence of their practical meaning and implications for sustained and effective clinical practice based on them has yet to be determined. In contrast, board certification and higher scores have been linked to higher-quality care among practicing physicians. ^23-26 The relationship between these measures and quality of care is complex and many physicians may perform well on quality measures despite not achieving high academic scores. New assessments and methodologies could be developed to better capture preparedness for practice.

Our study is subject to multiple limitations. First, the study period is prior to full implementation of the single accreditation system, and our results may not apply to those programs formerly solely accredited by the American Osteopathic Association (AOA). Second, the milestones have been updated since our study period and our findings may not translate to the new milestones. Third, the milestones themselves have limitations in assessing resident performance and may be better suited to track intraprogram than interprogram performance. ^{18, 19} Finally, despite the use of accepted academic measures and controlling for clustering of outcomes, there are likely other resident, residency program, institutional, and community characteristics that influence academic progression for which we were unable to control in our study.

In conclusion, using a large, longitudinal sample of family medicine residents we found statistically significant, though small, differences in assessments of medical knowledge and milestones between rural and urban training sites. The implications of these findings in judging the quality of rural programs are much less clear and warrant further study, including their impact on rural patient outcomes and community health. These results do support providing rural programs the funding and specialized resources available in larger programs.

This research was supported by the Bureau of Health Workforce (BHW), Health Resources and Services Administration (HRSA), US Department of Health and Human Services (HHS) under cooperative agreement #UH1HP29966. The information, conclusions and opinions expressed in this presentation are those of the authors and no endorsement by BHW, HRSA, or HHS is intended or should be inferred.

This study was recently presented at the following venues: National Rural Health Association, AAMC Group on Diversity and Inclusion and Health Workforce Research Joint Conference, The RTT Collaborative Annual Meeting, and AcademyHealth Annual Health Research Meeting.