Using the Family Medicine National Graduate Survey to Improve Residency Education by Monitoring Training Outcomes

Family medicine program requirements have required residencies to survey their graduates to assess outcomes. Prior to 2016, these surveys were typically institutional or regional, lacked comparable data, and had low response rates. The Family Medicine National Graduate Survey (Graduate Survey) was developed by the Association of Family Medicine Residency Directors (AFMRD) and the American Board of Family Medicine (ABFM) to provide programs more consistent, reliable feedback from their early-career graduates. The goal of the Graduate Survey is to provide programs with useful feedback and increase the specialty’s capacity to improve preparation for practice.1 The process of creating the survey has been previously published.2

Beginning in 2016, the Graduate Survey has been administered to ABFM diplomates 3 years after residency graduation via their online ABFM physician portfolio. Once data are collected, they are aggregated at the program level and shared with residencies, along with national comparison data, via each residency’s ABFM Resident Training Management portal. To protect diplomate confidentiality, residencies are only provided program-specific data if three or more graduates completed the survey. National-level reports are available on the ABFM website.3 An AFMRD/ABFM oversight committee has met annually to review survey performance and monitor content for currency and continued relevance. The conduct of the survey is primarily for programmatic evaluation and was deemed to not require institutional review board review; however, use of the data secondarily for research has received institutional review board approval. Researchers may request deidentified data from the ABFM, subject to review.

From 2016 to 2019, the response rate has been 66.7% to 73.6% (Table 1). The number of programs with eligible graduates has increased from 439 to 460, with 85.2% to 90.9% of programs receiving a program specific report on their graduates each year. Nearly two-thirds of programs with eligible graduates in all 4 years received four reports (Table 2). An additional 20.9% received three reports, 12.3% received one or two reports, leaving only seven programs (1.5%) that have yet to receive a program-specific report.

In each year, nonrespondents were more likely to be DOs, international medical graduates, and male, but these differences have been minimal and not statistically significant (Table 3). However, international medical graduates have increasingly not responded to the survey, which may further limit residency reports for programs with high numbers of such trainees.

Research studies using data from the Graduate Survey have shown a large gap between practice and preparation4 with further variation between academic- and community-based programs.5 Other studies using the data have shown lower odds of burnout with broader scope of practice,6 state-level variation in burnout,7 associations between residency training and buprenorphine prescribing,8 barriers to practicing obstetrics,9 provision of contraceptive services and abortion care,10,11 and participation in loan repayment programs.12

With multiple years of data, residencies have the capability to identify persistent gaps in their curricula and make changes. These data may also inform training outcomes on a larger scale by connecting with other data sources to investigate associations between program characteristics, self-assessed preparation for practice, quality and claims-based outcomes, and success on continuous certification.13,14 Longitudinal analyses have the potential to improve individual program and physician performance. For example, early family medicine Milestones data found that a relatively lower proportion of family medicine residents graduate at level 4 (ie, proficiency) in the systems-based practice Milestone 3 (advocates for individual and community health) than others.15 Examining this finding in the context of what graduates are actually doing in practice may help inform curriculum changes and assessment around a particular subcompetency and Milestone.

Feedback to residencies could be enhanced with even more data. The ABFM collects practice intentions and satisfaction with training when residents register for their initial certification examination.16 Aggregate reports of these data could be made available to programs to monitor intentions and practice. The ABFM collects other data later in a diplomate’s career which could also be fed to residences to track outcomes even farther from graduation. Medicare claims data may offer a window into practice by providing data on all graduates on comprehensiveness, continuity, costs of care, and low-value care.

Data from the Graduate Survey could also be repackaged to meet the needs of other stakeholders, while still respecting respondent confidentiality. State-based organizations such as academies of family medicine often advocate for graduate medical education (GME) training expansion and practice incentive programs, and graduate survey data on how many graduates remained in state, worked in underserved settings, or provide care in specific areas may provide the outcomes data needed to further those efforts.17 If residencies were willing to be identified, summary data could be used for improvement efforts on outcomes between programs with disparate outcomes. Medical students could use data to guide their residency selection by seeing which programs produce graduates with the practice they desire.

Using data from the survey, each residency can determine if it is meeting its mission and goals. For residencies with a mission to produce physicians for underserved areas or populations, practice addresses are geocoded and linked to rural status and practice in a Health Professional Shortage Area. Residencies with strong procedural or obstetrics focus can also track if their graduates are applying these skills in practice.

While the Graduate Survey has provided data to residencies to improve residency education, there are limitations to the methodology. First, since the ABFM administers the survey to its diplomates, recent graduates who either certify with other boards or do not certify are not included. This issue will be exacerbated with the single accreditation system and large numbers of programs whose graduates may largely certify with the American Osteopathic Board of Family Physicians (AOBFP). Collaboration with the AOBFP on combined delivery of a survey to their diplomates would enhance the utility of the survey. Second, in order to protect respondent confidentiality, three or more graduates must respond for a program to receive a report. The ABFM is working on a dashboard that would allow pooling of data over years to ensure all programs get reports. Finally, the content of the survey is broad to ensure generalizability, but may lack detail specific to some residencies’ mission and needs.

In conclusion, the Graduate Survey provides a model for partnership of certifying boards and the GME community to create meaningful measures and feedback to residencies.13 Data from the survey has the potential to improve residency education, track trends in the delivery of care by early-career family physicians, and promote social accountability of GME funds.