Family Medicine Resident Remediation Compared Across Two CERA Studies

Remediation is frequently necessary to ensure competency to practice family medicine independently upon residency graduation. Despite the immense time and monetary costs of remediation to residency programs, a great need remains to ascertain who will need remediation, how to remediate, how long to remediate, and what factors lead to successful remediation outcomes. Neither program directors nor residents themselves can reliably predict who may need remediation. ^{1, 2} Some consistent factors associated with medical student applicants who have difficulties in residency include prior issues with professionalism, exam failures, longer times between medical school and residency, and extension of medical school. ³ Other risk factors associated with the need for remediation are numerous and, at times, contradictory; these include residency setting and applicant characteristics such as gender, age, medical school location, type of medical degree, race, marital status, and transfer resident status. ^3- 7 Meanwhile, residency program factors that increase the likelihood of needing remediation include high resident workload, stress, fatigue, and burnout. ^{3, 7} Additionally, frequently needing to adapt to new clinics, hospitals, and cities has been associated with increased burnout and subsequent need for remediation. ^{3, 7} 

Whereas prior studies have noted medical knowledge as the primary reason for remediation in family medicine and internal medicine residents, professionalism issues have become the leading cause for remediation in family medicine residencies. ^{1, 4, 8-10} According to the 2017 Council of Academic Family Medicine Educational Research Alliance (CERA) data, 93% of programs had at least one resident requiring remediation in the prior 3 years. ⁴ The frequency of remediation for family medicine residents was reported to be between 2% and 15%. ^{5, 8- 12} Approximately 79% to 91% of residents have successfully remediated, with a mean remediation time ranging from 3 months to 1 year. ^{4, 9, 12} Unfortunately, 47% of remediations resulted in repeat remediations, while 18% have led to probations. ⁹ In 2018, the approximate cost to a surgical program for faculty, resident, and administrative time was estimated between $3,400 and $5,300 per remediation, with total faculty time per remediation being between 15.65 and 20.68 hours. ¹³ However, this cost estimate included remediation of technical skills, which was the most expensive to remediate. ¹³

During the COVID pandemic from 2020 to 2023, many medical school students’ clinical experiences transitioned from in-person learning to online electives. The pandemic influenced teaching environments and impacted learners’ educations. The Accreditation Council of Graduate Medical Education (ACGME) requires all family medicine residency programs to train and assess residents in six core competencies: medical knowledge, patient care, interpersonal communication skills, practice-based learning and improvement, system-based practice, and professionalism. In 2017 and 2023, CERA surveyed family medicine program directors across the United States regarding remediation prevalence, training of faculty members for remediation, tools for remediation, and factors associated with remediation. Whether findings from the 2017 survey would differ from those from the survey following the pandemic was unclear.

The objective of this study was to evaluate the factors associated with remediation in the 2023 CERA study and to examine any association between the faculty and program director (PD) experience, training in remediation, and the program didactic content with the identification for remediation, the core competency most frequently remediated, the number of residents undergoing remediation, the percentage of residents remediating successfully, and the most common duration of remediation efforts. We also compared changes in remediation practices between the 2017 and 2023 CERA studies. A comparison of CERA data at these two points in time allowed for observations of remediation trends across the ACGME core competencies. The authors hypothesized that the 2023 CERA study would reflect an increased need for remediation of medical knowledge.

The CERA survey is administered twice annually to all ACGME-accredited US family medicine PDs identified through the Association of Family Medicine Residency Directors (AFMRD). The survey includes recurring questions about PD and residency characteristics such as PD years of experience, gender, number of total residents, type of residency program, community size, and US region of the residency program. Ten retrospective questions were included to investigate areas related to residency remediation. Six of the 10 survey questions about remediation were derived from the 2017 CERA survey. ¹⁴ The CERA steering committee reviewed and pretested the four new questions with family medicine educators who were separate from the target population. The 2017 and 2023 CERA survey questions can be found on the Society of Teachers of Family Medicine (STFM) website and in Supplemental Material 1 and 2, respectively.

Following the review and approval by the American Academy of Family Physicians Institutional Review Board, the 2023 survey was distributed to ACGME-accredited US family medicine residency PDs from September 26, 2023, to October 30, 2023, via SurveyMonkey (SurveyMonkey Inc). Three follow-up emails were emailed weekly to encourage nonrespondents to participate, with a fourth reminder sent 1 day before survey closure. Of the 754 PDs listed by AFMRD, 715 were eligible to participate.

A total of 39 PDs were excluded from the current study: 10 had no functional email address available, and 29 had fewer than three resident classes. Responses were received from 278 eligible programs. Seven surveys were removed because only the first survey question was completed. The results of the 2017 CERA survey for program directors have been published previously. ⁴

Data from both surveys were analyzed using SPSS version 29 for Windows (IBM). Descriptive statistics, two-sided independent t tests, and Pearson ꭓ² tests were applied to the survey responses from the 2017 and 2023 CERA surveys. The raw data analyzed for this study are available to members of STFM, the North American Primary Care Research Group, and the Association of Departments of Family Medicine at the URL https://www.stfm.org/publicationsresearch/cera/pasttopicsanddata/pastsurveyaudience/#tab-6395.

The 2023 CERA survey of program directors had a 37.9% response rate. Demographic data from the 2017 and 2023 CERA surveys are displayed in Table 1. Program type, location, community size, and total years as PD were not statistically different between 2017 and 2023. However, a statistically significant increase in the number of female program director respondents occurred between 2017 and 2023 (41.9% and 53.7%, respectively; P=.0002).

Most PD respondents reported 1 or 2 residents (39.5%) or 3 or 4 residents (36.8%) needing remediation in 2023. This trend was apparent in 2017 as well (36.7% and 34.1%, respectively; Figure 1). The likelihood of successful remediation of 76% or more residents increased significantly from 2017 to 2023 (P=.006), while the percentage of programs able to successfully remediate 100% of their residents remained stable (P=.08; Figure 2).

In 2017, residents usually were first recognized as needing remediation during the first half of the intern year (46%). However, in 2023, identification occurred significantly more often during the second half of the intern year (56.1%; P<.001 and P<.001, respectively). The typical time frame from the start of remediation to successful completion was 6 to 12 months (47.5% in 2017, 50% in 2023; P=.57) in both groups. Successful remediation required more than 12 months for 5.7% of residents in 2023 and 8.7% of residents in 2017 (P=.18). None of the factors evaluated were associated with the length of remediation needed (Table 3).

Figure 3 illustrates the top ACGME core competency needing remediation during the prior 3 years and compares the 2017 and 2023 CERA survey results. The top ACGME core competency needing remediation remained professionalism in both surveys (38.1% in 2017, 45.1% in 2023; P=.10) with medical knowledge being the second most common (30.2% in 2017, 25.2% in 2023; P=.20;Figure 3).

The deficiency most often identified during the first half of the intern year was interpersonal and communication skills. Having more difficulty with practice-based learning and improvement was most often associated with larger community size (P=.04), while residency location (East/Central) was associated with medical knowledge deficits (P=.006). If a program director responded that medical knowledge deficits were the most common ACGME core competency needing remediation in the program, that factor also was associated with five or more residents undergoing remediation (P=.015). The percentage of non-US graduates and the number of residents undergoing remediation were related to patient care being identified as a main concern for remediation (P=.046 and P=.044, respectively). Programs with more didactic hours reported more frequent remediation for interpersonal and communication skills (P=.04). No factors were associated with identifying systems-based practice issues as the main ACGME core competency identified for resident remediation.

The factors significantly associated with successful remediation of all residents included the most frequent resident ACGME competency deficit being medical knowledge (P=.01) or patient care (P=0.02). Moreover, programs with fewer residents requiring remediation were more likely to effectively remediate 100% of their residents in difficulty (P<.001). When medical knowledge deficits were discovered in the first half of the intern year, the resident was more likely to successfully remediate and complete residency (P=.003).

Variables that were not associated with the percentage of residents who successfully remediated included the type of residency program (university, community-based), region of the US, community size, number of residents, when the need for resident remediation was identified, the number of residents matched in the Supplemental Offer and Acceptance Program (SOAP), or the number of non-US graduates. Neither the number of SOAP residents nor the percentage of non-US graduates was associated with the number of residents needing remediation.

In 2023, most program directors noted that less than 25% of their core faculty received specific remediation training (62.9%), such as participating in the STFM Residency Faculty Fundamentals Certificate program. Only five programs (1.9%) reported that all their core faculty had completed specific remediation training. Less specialized remediation training was reported among faculty at university-based facilities and outside of the Midwest. Instead, having a higher percentage of faculty with specialized remediation training was associated with a higher number of hours devoted to teaching professionalism skills (P=.001) and the level of remediation training obtained by the PD (P<.001). The percentage of faculty with specialized remediation training was not associated with the number of residents requiring remediation, the percentage of successful remediation, when the need for remediation was identified, or the number of matches in the SOAP.

Of PD respondents, 71.8% had moderate to substantial remediation training, such as attending a lecture or a workshop. Six percent of respondents reported having had no training in remediation, whereas 19.2% reported having only read materials or discussed remediation with colleagues. From 2017 to 2023, the percentage of PD respondents noting a substantial amount of training, including attending a workshop or doing fellowship training, did not change (37.0% and 32.7%, respectively; P=.30). However, no significant differences were found across the 2017 and 2023 data related to the reported amounts of remediation training. Higher levels of remediation training by residency PDs were associated with more residents needing remediation (P=.004) but not with residency location or community size.

PD characteristics such as gender, race, degree type (MD, DO), self-identifying as underrepresented in medicine, or years of experience were not associated with the number of residents requiring remediation, the timing of remediation identification, or the likelihood of successful resident remediations. A greater number of years as a program director was associated with listing patient care as the top ACGME competency requiring remediation (P=.009) but not the level of remediation training. No association appeared between factors such as the amount of remediation training or the number of non-US graduates.

Regarding teaching professionalism, most programs (40.8%) set aside 1 to 3 hours per year to teach these skills, while another 35.5% of the programs offered 4 to 6 hours yearly. Notably, 12% of PDs reported 10 or more hours each year to teach professionalism. Programs with a larger number of residents requiring remediation trended toward having more professionalism skills training; however, this finding did not meet statistical significance (P=.052). While professionalism skills training is typically held for less than 6 hours per year, most residencies (78.4%) have 4 to 6 hours of scheduled didactics per week, with a few programs (1.9%) having 13 or more hours per week. Neither the hours of professionalism training nor weekly didactics were associated with the number of residents requiring remediation, when the need for remediation was identified, or the percentage of residents who could be successfully remediated.

Consistent with prior studies, the most frequently cited ACGME core competencies needing remediation continue to be professionalism followed by medical knowledge. ⁴, ⁸ Given the increased distance learning and decreased direct patient contact hours, residents from 2020 to 2023 scored roughly 1.25 years behind their former peers on the in-training exam. ¹⁵ Despite this potential challenge for PDs, we found no significant difference in the frequency of medical knowledge remediation reported by the PDs in the 2023 survey. Nonetheless, programs may have adjusted their threshold for formal remediation of medical knowledge and/or changed their didactics to accommodate the aforementioned variables, thereby leading to later initiation of remediation for medical knowledge. ¹⁵ Alternatively, lack of training in identifying residents in need, hesitancy in classifying a resident as needing remediation for legal and potential career consequences, and limited understanding of how to remediate residents may have affected the likelihood of requiring residents to remediate. ¹⁶

Some concerning discoveries were made when assessing data between the 2017 and 2023 CERA survey of PDs; the percentage of PD respondents with little to no remediation training remained around 25%. This lack of change may be related to the frequent turnover of PDs, as evidenced by most respondents in the current survey not having been a PD at the time of the 2017 survey. However, we found no association between the highest level of remediation training and years as a PD. Other factors may include the monetary and opportunity costs associated with attending programs focused on resident remediation. Nevertheless, we observed no differences across gender, race, region, community size, or residency type compared to the highest level of PD or core faculty remediation education. Fortunately, multiple resources for remediation have been created since the 2017 data were published, including the STFM Residency Faculty Fundamentals Certificate Program as well as two free online resources: STFM Competency-Based Medical Education Toolkit for Residency Programs ¹⁷ and the ACGME Remediation Toolkit. ¹⁸

We speculated that multiple areas might be associated with the timing of remediation, likelihood of successful remediation, and duration of remediation. These crucial outcomes had no significant association between hours of didactics, hours of professionalism training, the number of residents matched in the SOAP, percentage of non-US graduates, faculty or PD remediation training, program characteristics, or PD demographics.

Identifying residents for remediation in the second half of the intern year occurred more frequently in 2023, whereas in 2017, most residents would start remediations in the first half of the intern year. Of note, the percentage of successfully remediated residents increased from 2017 to 2023, while the time needed for successful remediation remained unchanged. We speculated that this effect may have been due to residents having less severe deficits in professionalism, leading to later identification and more successful remediation. Alternatively, as we have personally experienced, programs may have intuitively given learners more time to remedy their deficits instead of requiring formal remediation.

The strengths of the study include diverse programs in terms of the number of total residents, the number of residents matched in the SOAP, the percentage of non-US graduates, community sizes, and PD characteristics. In turn, this study provides nationwide/regional benchmarks for remediation type (typically professionalism and medical knowledge), duration (usually 6–12 months to remediate successfully), and number of residents (1-4 every 3 years) across numerous US residency settings as well as references for hours of didactic and professionalism training across programs.

Limitations of the current study included having a response rate of 37.9% and relying on indirect reports from PDs rather than accessing data directly, which introduces recall bias secondary to the primacy and/or recency effect, social desirability bias, and selection bias. The cross-sectional design offered glimpses of information limited to only a specific period. Although the demographics in 2017 and 2023 were similar, the resident complements and PDs were dynamic groups. The study presented mostly ecological-level data. Furthermore, given the study’s retrospective nature, one cannot conclude causation, assume equal representation of the population at-large, or apply the data to populations not directly studied, such as residents in other countries or different medical specialties. How having so many responses from new PDs would affect survey responses needs to be clarified. Additionally, numerous variables were run against one another to look for significant differences. As a result, some of the significant findings we discovered were likely due to chance. Just as in the current study, multiple dilemmas are evident and ill-defined in both remediation research and practice: how to determine the need for remediation, how remediation is defined (eg, informal, formal, additional reading assignments; attending a class; following a performance improvement plan, having an individualized learning plan), and what determines successful completion. ¹⁹ These determinations can be subjective and, as a result, prone to bias and discrimination.

Areas for further research or research dissemination include identifying recruitment practices, residency factors, and remediation plans/processes associated with 100% remediation success and/or preventing the need for remediation. Additional areas of investigation include the prospective evaluation of individualized learning plans in competency-based medical education (CBME) and how to effectively train faculty and PDs on CBME practices to enhance success among their residents. ²⁰

Professionalism was the top core competency needing remediation among family medicine residents in both the 2017 and 2023 CERA surveys of PDs. No associations were present between resident or program characteristics, faculty training in remediation, or the core competency requiring remediation and the number of residents needing remediation or the percentage of residents successfully completing remediation. PD factors were not associated with any differences in remediation outcomes.