A Wealth of Data, a Paucity of Outcomes: What Can We Learn From the ACGME Accreditation Data System?

Clinical experiences in residency have a powerful impact on preparation for autonomous practice. Future practice patterns and scope of practice are both heavily influenced by the clinical training environment to which a resident is exposed.1 Several years ago, the Review Committee for Family Medicine influenced the residency practice environment by requiring that family medicine residencies ensure their programs place additional emphasis on the role modeling and volume of exposure for pediatric, hospital, and maternity care, among other things.2 What has been the impact of those requirements on residency practice?

The Accreditation Council for Graduate Medical Education (ACGME) collects specialty-specific data each year on all programs through the Accreditation Data System (ADS). These data are tied to key clinical educational experiences required for residency training. As this is required for accreditation, the data set is a robust and consistent representation of the entire family medicine GME community. Data required from family medicine programs includes resident continuity visits, family medicine practice (FMP) demographics, delivery numbers for total vaginal and continuity deliveries per graduate, the most common diagnoses seen by residents in the ambulatory and adult inpatient and pediatric inpatient settings, as well as the average daily inpatient load for residents in those settings. The most commonly performed and required procedures are also listed. The corresponding trends for this data in family medicine residency practices over the last 10 years will be critical in informing the writing of future program requirements. It should be noted that national FM ADS data dating back to 2008 are available online in the ACGME data book archives, or upon request from the ACGME ADS data analytics division.

The ACGME publishes the aggregated data for all family medicine programs each year in a national report, made available to program directors and designated institutional officials.3 Surprisingly, these robust data sets have not been routinely analyzed or tracked for trends or graduate outcomes. In fact, very few references can be found that cite these national ADS reports. The upcoming major revision process has spurred a more thorough review of the last 10 years of ADS data, and trends in citations issued by ACGME Review Committee for Family Medicine (ACGME RC-FM) and some of the more notable trends have been published in the briefs leading up to the national summit on the major program revisions.4,5

Data trends for mean FMP continuity visits by graduating residents showed declines over the last decade, from a peak of 1,864, to 1,717 most recently (-7.8%) during a time when ambulatory training emphasis has been advocated (Figure 1). The most precipitous drop in the last year is due to the impact of the pandemic on in-person care. Decreases in resident FMC pediatric visits (defined as age <10 years, based on current program requirements), down 3.3% over 10 years from a peak of 15.6% to 12.3%, mirror national trends. In order to correlate with American Board of Family Medicine (ABFM) graduate survey data that asks for percentage of patients seen under age 5 years and under age 18 years, the percentage of visits in FMC by residents up to age 19 years was also reviewed and showed a decline of 5.1% over the last 10 years (25.4% to 20.3%).

The mean percentage of resident FMC visit with elders (defined as age 60 years) has correspondingly increased over the same time period by 6.4%, from a nadir of 16.9% to a peak of 23.3% most recently, which may reflect our aging population and shrinking pool of general internists. The mean number of vaginal deliveries by graduating residents has declined by 30%, from mean of 55.6 to 38.8, over the last 10 years, driven in large part by declines in the mean number of continuity deliveries by graduates over that same span, from 12.8 to 8.8 (down 31%; Figure 2). This may be tied to changes in program requirements that eliminated numerical requirements for deliveries in 2013 and decline in US birth rate. However, scatterplot analysis indicates there are a significant number of programs that can achieve numbers that are above the declining national average. The type of procedural training experiences remained fairly consistent over the last 10 years and mirrored scope of practice data provided by ABFM graduate survey respondents. The case mix of most common diagnoses seen in FMPs was also fairly consistent over time, with some modest increases in the frequency of chronic conditions seen and modest declines in prenatal care over the last decade. Hospital diagnoses seen by residents for adults and children remained consistent over the last decade with some minor variations in rank order of frequency.

We include histograms for the resident visit and vaginal delivery data (Figures 1 and 2), these include the most recent data from 2020, which were heavily influenced by the pandemic in the last quarter of the academic year. The modest decline in resident continuity visits prior to the pandemic translates into roughly 150,000 lost continuity visits annually for the entire graduating cohort of FM residents. The clinical significance of these lost visits is unclear.

So, what does all this data tell us? In many respects, it conveys that the adage “the clinic is the curriculum” holds true, in that the ambulatory, continuity experiences of residents in FMPs is consistent with the practice patterns seen amongst practicing physicians.6 Declining maternity care, pediatric visits, and increasing multimorbidity elder care are trends seen nationally outside of residencies.7,8 While there is some correlation with the decline in reported deliveries in residencies and the removal of requirements for total deliveries by graduates, the other trends indicate that forces outside of the requirements have just as much, if not more influence on residency practice patterns. Competency is often a result of repetition in many situations, and in this instance the repetition is the clinical experience, whether a continuity visit or a delivery. What constitutes the minimum range to achieve such competency is difficult to say from these data. However, comfort with scope of practice after graduation may be a surrogate marker, and ABFM graduate surveys do hint that scope is shrinking in those areas where we see volumes declining, such as pediatrics and maternity care.9 The key question here is: does the known imprint of training more strongly influence the practice of the graduate, or have the changing demographics of the practice environment influenced the clinical experience of residents? We suspect it is an entanglement of both. For example, a graduate of a program located in a community with a high density of pediatric clinicians and specialists may not feel competent or confident to care for children due to a low volume of continuity pediatric clinical experience seen in their training or lack of such role modeling by the core faculty of the program. These issues must be addressed through systematic, regular analysis of the clinical experiences of residents by the program, with the determination to improve areas of clinical opportunity to achieve competence.

Yet, the data set also reveals that residency practice sites are islands of broader scope, often in a surrounding sea of more uniform ambulatory primary care dictated by health systems and physician lifestyle choices. Residencies continue to showcase practices where maternity care, hospital care, and procedural care are required elements of the FMP. Residencies are incubators of new innovative practice areas such as the use of point-of-care ultrasound in procedural care. Trends in citations issued by ACGME RC-FM provide another source of insight into resident patient experiences. What we do not know from these data sources are what patient or community outcomes result from this ongoing role modeling. As we begin to embark on the process of major program revisions, the primary end goal is to achieve better health outcomes for our population through the rigorous training of competent family physicians. Can we find a way to wed the power of national data collected in ADS with these desired outcomes of residency training? If we can, it may pave the way for a powerful message on the value of training more residents in family medicine to improve the health of our nation.