National Family Medicine Residency Outcomes Project Methodology

Family medicine is re-envisioning how to train residents.^1,2 Programs need to determine how to innovate and implement new Accreditation Council for Graduate Medical Education (ACGME) Program Requirements.³ To identify best practices that produce desired graduate outcomes, detailed information about residency curriculum needs to be linked to graduate outcomes. The current state of family medicine curricular research consists of mostly small studies with limited generalizability or studies using American Board of Family Medicine (ABFM) data that do not have detailed data about programs.^4-9 The Council of Academic Family Medicine (CAFM) Education Research Alliance (CERA) and ABFM National Family Medicine Residency Outcomes Project (FM-ROP) is designed to provide this information. This report describes the FM-ROP methodology.

This prospective cohort study followed the 2018 family medicine residency graduates into practice until they completed the 2021 National Graduate Survey (NGS).

Exposures during residency were collected via a special CERA program director survey. Items were developed through the standard CERA process^10,11 and additional items were added by an FM-ROP steering committee.  

The CERA steering committee evaluated questions for consistency with the overall project aims, readability, reliability, and validity. Pretesting was done with experienced family medicine educators not in the target population. Questions were modified following pretesting for flow, timing, and readability. The 2018 CERA Special Program Director (PD) Survey is publicly available.¹⁵

The sampling frame for the CERA survey was all ACGME-accredited US family medicine residency program diretors as identified by the Association of Family Medicine Residency Directors in the summer of 2018. Data were collected from July to October 2018. Email invitations were delivered with the survey utilizing SurveyMonkey (SurveyMonkey Inc, San Mateo, CA). A total of nine follow-up emails to encourage nonrespondents to participate (eight sent weekly plus a final reminder 48 hours before closing) were sent after the initial email invitation. The survey was emailed to 617 eligible program directors; 31 emails could not be delivered. The final sample size was 586. 

The NGS is administered to ABFM diplomates 3 years after graduation^12,13 with questions and results available online.¹⁴ The 2021 survey opened in January 2021 and was to close in December but was kept open through June 2022 due to ABFM programmatic changes due to COVID-19. Nonrespondents were sent up to six reminders. Resident demographics were gathered from ABFM administrative databases. Race was self-reported by the resident during registration for the 2018 ABFM initial certification examination in response to a “select best” question.

The program director responses in the 2018 CERA survey were merged with the residency graduate responses on the NGS by ACGME number. The unit of analysis was the residency graduate. The final FM-ROP data set contains all respondents to the NGS whose PD responded to the CERA survey.

We used t tests and  χ² tests to assess differences between groups surveyed and included in our final analytic cohort. First, we assessed the representativeness of the final graduate sample after merging the two data sets with the overall population of 2018 residency graduates. We further assessed the representativeness of the programs by using ABFM administrative data to calculate size of the program, the region, and the average initial certification examination score and testing for significance in distribution between receiving a survey, responding, and having a responding graduate. We used SAS v9.4 software (SAS Inc, Cary, NC) for all analysis. The overall project was approved by the American Academy of Family Physicians Institutional Review Board.

The response rate for the 2018 CERA survey was 43.7% (255/586). The response rate for the 2021 NGS was 45.0% (1,623/3,610). After merging the two data sets, 44 programs were lost due to no NGS data and 844 graduates due to no CERA data. Our final analytic sample included 779 graduates from 211 residencies (Figure 1).

Table 1 compares the demographics of those graduates eligible for the NGS to those included in the ultimate FM-ROP cohort. There were statistically significant but nonmeaningful differences in age and race.

Table 2 compares the characteristics of programs with graduates included in the FM-ROP cohort to all the programs that received a CERA PD survey. There were no regional differences or differences in ABFM examination scores between responders and nonresponders, but a greater number of larger programs had graduates included in the FM-ROP cohort. Table 3 further describes the training program demographics. Not surprisingly given the survey is of ABFM diplomates, few graduates from former American Osteopathic Association-accredited programs are represented in this sample. Program size changes significantly when the unit of analysis is the graduates and not the programs, and since there are more individual graduates from the larger programs, larger programs are represented at a higher percentage when focusing on graduates

This is the first national cohort study in any specialty that follows an entire class of residency graduates by assessing program director perspectives on their training, including program structures and processes, then linking this to graduate survey data that includes postgraduation practice patterns and outcomes. New ACGME program requirements allow greater flexibility in processes and curricula. Programs will need objective evidence to best evaluate the changes and choices they make so that they can achieve desired graduate outcomes. FM-ROP is an important step in this process.

Limitations include the low response rate which may reduce generalizability. The ABFM NGS response rate was lower than anticipated likely due to it being administered during the COVID-19 pandemic. The CERA program director survey response rate is similar to others during this time and reflects the challenges of surveying physicians. Additionally, the sampling frame is of ABFM-certified family physicians, and does not capture uncertified physicians or those certified by the American Osteopathic Board of Family Physicians. While the program director survey was comprehensive, there are likely unmeasured residency factors that affect graduate outcomes. Finally, program director responses may be subject to social desirability bias and may report curricula items that may only be nominally present.

FM-ROP is the first national study to evaluate graduate outcomes for one class of residency graduates. Findings from analyses of these data should provide actionable data to PDs. This methodology could be implemented in other specialties.