— COMMENTARIES —

Using the Family Medicine National Graduate Survey to Improve Residency Education by Monitoring Training Outcomes

Lars E. Peterson, MD, PhD

Fam Med. 2021;53(7):622-625.

DOI: 10.22454/FamMed.2021.719992

What Is the Graduate Survey?

Family medicine program requirements have required residencies to survey their graduates to assess outcomes. Prior to 2016, these surveys were typically institutional or regional, lacked comparable data, and had low response rates. The Family Medicine National Graduate Survey (Graduate Survey) was developed by the Association of Family Medicine Residency Directors (AFMRD) and the American Board of Family Medicine (ABFM) to provide programs more consistent, reliable feedback from their early-career graduates. The goal of the Graduate Survey is to provide programs with useful feedback and increase the specialty’s capacity to improve preparation for practice.1 The process of creating the survey has been previously published.2

Beginning in 2016, the Graduate Survey has been administered to ABFM diplomates 3 years after residency graduation via their online ABFM physician portfolio. Once data are collected, they are aggregated at the program level and shared with residencies, along with national comparison data, via each residency’s ABFM Resident Training Management portal. To protect diplomate confidentiality, residencies are only provided program-specific data if three or more graduates completed the survey. National-level reports are available on the ABFM website.3 An AFMRD/ABFM oversight committee has met annually to review survey performance and monitor content for currency and continued relevance. The conduct of the survey is primarily for programmatic evaluation and was deemed to not require institutional review board review; however, use of the data secondarily for research has received institutional review board approval. Researchers may request deidentified data from the ABFM, subject to review.

Results

From 2016 to 2019, the response rate has been 66.7% to 73.6% (Table 1). The number of programs with eligible graduates has increased from 439 to 460, with 85.2% to 90.9% of programs receiving a program specific report on their graduates each year. Nearly two-thirds of programs with eligible graduates in all 4 years received four reports (Table 2). An additional 20.9% received three reports, 12.3% received one or two reports, leaving only seven programs (1.5%) that have yet to receive a program-specific report.

In each year, nonrespondents were more likely to be DOs, international medical graduates, and male, but these differences have been minimal and not statistically significant (Table 3). However, international medical graduates have increasingly not responded to the survey, which may further limit residency reports for programs with high numbers of such trainees.

Research studies using data from the Graduate Survey have shown a large gap between practice and preparation4 with further variation between academic- and community-based programs.5 Other studies using the data have shown lower odds of burnout with broader scope of practice,6 state-level variation in burnout,7 associations between residency training and buprenorphine prescribing,8 barriers to practicing obstetrics,9 provision of contraceptive services and abortion care,10,11 and participation in loan repayment programs.12

Possible Uses and the Future

With multiple years of data, residencies have the capability to identify persistent gaps in their curricula and make changes. These data may also inform training outcomes on a larger scale by connecting with other data sources to investigate associations between program characteristics, self-assessed preparation for practice, quality and claims-based outcomes, and success on continuous certification.13,14 Longitudinal analyses have the potential to improve individual program and physician performance. For example, early family medicine Milestones data found that a relatively lower proportion of family medicine residents graduate at level 4 (ie, proficiency) in the systems-based practice Milestone 3 (advocates for individual and community health) than others.15 Examining this finding in the context of what graduates are actually doing in practice may help inform curriculum changes and assessment around a particular subcompetency and Milestone.

Feedback to residencies could be enhanced with even more data. The ABFM collects practice intentions and satisfaction with training when residents register for their initial certification examination.16 Aggregate reports of these data could be made available to programs to monitor intentions and practice. The ABFM collects other data later in a diplomate’s career which could also be fed to residences to track outcomes even farther from graduation. Medicare claims data may offer a window into practice by providing data on all graduates on comprehensiveness, continuity, costs of care, and low-value care.

Data from the Graduate Survey could also be repackaged to meet the needs of other stakeholders, while still respecting respondent confidentiality. State-based organizations such as academies of family medicine often advocate for graduate medical education (GME) training expansion and practice incentive programs, and graduate survey data on how many graduates remained in state, worked in underserved settings, or provide care in specific areas may provide the outcomes data needed to further those efforts.17 If residencies were willing to be identified, summary data could be used for improvement efforts on outcomes between programs with disparate outcomes. Medical students could use data to guide their residency selection by seeing which programs produce graduates with the practice they desire.

Using data from the survey, each residency can determine if it is meeting its mission and goals. For residencies with a mission to produce physicians for underserved areas or populations, practice addresses are geocoded and linked to rural status and practice in a Health Professional Shortage Area. Residencies with strong procedural or obstetrics focus can also track if their graduates are applying these skills in practice.

While the Graduate Survey has provided data to residencies to improve residency education, there are limitations to the methodology. First, since the ABFM administers the survey to its diplomates, recent graduates who either certify with other boards or do not certify are not included. This issue will be exacerbated with the single accreditation system and large numbers of programs whose graduates may largely certify with the American Osteopathic Board of Family Physicians (AOBFP). Collaboration with the AOBFP on combined delivery of a survey to their diplomates would enhance the utility of the survey. Second, in order to protect respondent confidentiality, three or more graduates must respond for a program to receive a report. The ABFM is working on a dashboard that would allow pooling of data over years to ensure all programs get reports. Finally, the content of the survey is broad to ensure generalizability, but may lack detail specific to some residencies’ mission and needs.

In conclusion, the Graduate Survey provides a model for partnership of certifying boards and the GME community to create meaningful measures and feedback to residencies.13 Data from the survey has the potential to improve residency education, track trends in the delivery of care by early-career family physicians, and promote social accountability of GME funds.

Acknowledgments

Eric Holmboe, MD, Karen Mitchell, MD, and Robert L. Phillips, Jr, MD, MSPH, contributed conceptually to an early draft of this paper.

Disclaimer: The views expressed in this article represent the personal opinions of the author and are not necessarily the views or policies of his institutions.

References

  1. Mitchell KB, Maxwell L, Miller T. The national graduate survey for family medicine. Ann Fam Med. 2015;13(6):595-596. doi:10.1370/afm.1874
  2. Weidner AKH, Chen FM, Peterson LE. Developing the National Family Medicine Graduate Survey. J Grad Med Educ. 2017;9(5):570-573. doi:10.4300/JGME-D-17-00007.1
  3. National family medicine residency graduate reports. American Board of Family Medicine https://www.theabfm.org/research/national-family-medicine-residency-graduate-reports. Accessed November 18, 2020.
  4. Peterson LE, Fang B, Puffer JC, Bazemore AW. Wide Gap between preparation and scope of practice of early career family physicians. J Am Board Fam Med. 2018;31(2):181-182. doi:10.3122/jabfm.2018.02.170359
  5. Carek PJ, Mims L, Kirkpatrick S, et al. Does community- or university-based residency sponsorship affect graduate perceived preparation or performance? J Grad Med Educ. 2020;12(5):583-590. doi:10.4300/JGME-D-19-00907.1
  6. Weidner AKH, Phillips RL Jr, Fang B, Peterson LE. Burnout and scope of practice in new family physicians. Ann Fam Med. 2018;16(3):200-205. doi:10.1370/afm.2221
  7. Hansen A, Peterson LE, Fang B, Phillips RL Jr. Burnout in young family physicians: variation across states. J Am Board Fam Med. 2018;31(1):7-8. doi:10.3122/jabfm.2018.01.170269
  8. Tong ST, Hochheimer CJ, Peterson LE, Krist AH. Buprenorphine provision by early career family physicians. Ann Fam Med. 2018;16(5):443-446. doi:10.1370/afm.2261
  9. Barreto TW, Eden AR, Hansen ER, Peterson LE. Barriers faced by family medicine graduates interested in performing obstetric deliveries. J Am Board Fam Med. 2018;31(3):332-333. doi:10.3122/jabfm.2018.03.170427
  10. Patel P, Narayana S, Summit A, et al. Abortion provision among recently graduated family physicians. Fam Med. 2020;52(10):724-729. doi:10.22454/FamMed.2020.300682
  11. Coffman M, Wilkinson E, Jabbarpour Y. Despite adequate training, only half of family physicians provide women’s health care services. J Am Board Fam Med. 2020;33(2):186-188. doi:10.3122/jabfm.2020.02.190293
  12. Nagaraj M, Coffman M, Bazemore A. 30% of recent family medicine graduates report participation in loan repayment programs. J Am Board Fam Med. 2018;31(4):501-502. doi:10.3122/jabfm.2018.04.180002
  13. Peterson LE, Carek P, Holmboe ES, Puffer JC, Warm EJ, Phillips RL. Medical specialty boards can help measure graduate medical education outcomes. Acad Med. 2014;89(6):840-842. doi:10.1097/ACM.0000000000000250
  14. Triola MM, Hawkins RE, Skochelak SE. The time is now: using graduates’ practice data to drive medical education reform. Acad Med. 2018;93(6):826-828. doi:10.1097/ACM.0000000000002176
  15. Hamstra SJ, Edgar L, Yamazaki K, Holmboe ES. Milestones Annual Report 2016. Chicago: Accreditation Council for Graduate Medical Education; October 2016. http://www.acgme.org/Portals/0/PDFs/Milestones/MilestonesAnnualReport2016.pdf. Accessed November 18, 2020.
  16. Peterson LE, Fang B, Phillips RL Jr, Avant R, Puffer JC. The American Board of Family Medicine’s data collection method for tracking their specialty. J Am Board Fam Med. 2019;32(1):89-95. doi:10.3122/jabfm.2019.01.180138
  17. Rockey PH, Rieselbach RE, Neuhausen K, et al. States can transform their health care workforce. J Grad Med Educ. 2014;6(4):805-808. doi:10.4300/JGME-D-14-00502.1

Lead Author

Lars E. Peterson, MD, PhD

Affiliations: American Board of Family Medicine, Lexington, KY | and Department of Family and Community Medicine, College of Medicine, University of Kentucky, Lexington, KY

Corresponding Author

Lars E. Peterson, MD, PhD

Correspondence: American Board of Family Medicine, 1648 McGrathiana Parkway, Suite 550, Lexington, KY 40511-1247. 859-538-7180.

Email: lpeterson@theabfm.org

Fetching other articles...

Mendeley

Loading metrics from Mendeley...

Loading the comment form...

Submitting your comment...

There are no comments for this article.

Related Content

Garvin RD, Carney PA. Residency Education Redesign: The Interplay of Innovation and Standardization. [published online ahead of print June 1, 2021]. Fam Med. https://doi.org/10.22454/FamMed.2021.862031.

Downloads & Info

Share