COVID-19 Impact on Family Medicine Residents Exam Performance

During the COVID-19 pandemic (2020-2022), several national surveys revealed that family medicine residency training had experienced disruption. ^1-3 Family medicine residencies are designed to train residents to develop the clinical skills to diagnose, manage, and treat a wide range of health conditions in various settings, including hospitals, outpatient clinics, and long-term care facilities. ⁴ However, most residents during 2020-2022 experienced changes to the content and structure of their education. For example, elective surgeries were canceled, and the number of outpatient visits decreased significantly. The traditional didactic and hands-on training styles between faculty and residents were impacted by efforts to reduce the risk of infection. ² Moreover, many family medicine residents were pulled into non-curriculum-based clinical services to care for patients admitted with COVID-19, which depleted mental and physical resources they could have used for independent learning. ⁵

Partly due to reduced learning opportunities, many professions observed decreased pass rates in their licensing examinations. For example, in nursing, the National Council Licensure Examination pass rate dropped more than 8 percentage points—from 88.2% in 2019 to 79.9% in 2022 for all first-time US-educated candidates. ⁶ The total pass rate for the Comprehensive Osteopathic Medical Licensing Examination of the United States dropped more than 3 points from 93.0% in 2019-2020 to 89.1% in 2021-2022 under the same passing standard. ⁷ While a decline in scores on the American Board of Family Medicine (ABFM) In-Training Examination (ITE) has been observed, how this decline has translated into the high-stakes Family Medicine Certification Examination (FMCE) is unclear. ⁸ In addition, we have not systematically quantified the learning loss. Previous research has shown that family medicine residents’ average score increase was 30-40 points per year of residency across all subgroups prior to the COVID-19 pandemic. ⁹ The goal of this study was to assess the magnitude of COVID-19’s impact on medical knowledge acquisition during residency, as measured by the ITE and FMCE.

The study's participants were residents who graduated from a family medicine residency accredited by the Accreditation Council for Graduate Medical Education (ACGME) and took the FMCE between 2017 and 2022. To ensure comparability of educational experience, this study included only residents who were enrolled in a 3-year program, did not transfer to another residency, completed residency on schedule, took the ITE in each year of residency, and took the FMCE in the first administration (spring) of their postgraduate year 3 (PGY-3). This study was approved by the institutional review board of the American Academy of Family Physicians. All statistical analyses were performed in R version 4.0.2 (R Foundation).

We obtained gender, country of medical school, and medical degree demographics from ABFM administrative data sets. Self-reported race and ethnicity data are collected from the demographics section of the Initial Certification Questionnaire (ICQ), which is completed 3 to 4 months prior to the examination. Gender was categorized as either male or female. Country of medical school was dichotomized into US/Canada and international. Medical degrees included only MD and DO. To be consistent with the US Census Bureau, race and ethnicity were considered separately. Specifically, ethnicity was dichotomized as Hispanic or Latino and non-Hispanic; respondents self-selected the single race category that best described themselves: Asian, White, Black, American Indian/Alaska Native, Native Hawaiian/other Pacific Islander, or Other.

Family Medicine Certification Scale

The Family Medicine Certification Scale is the common scale that is used to describe examinee performance on several of ABFM’s examinations, including the ITE and FMCE. Scores on this scale can range from 200 to 800 and are reported in increments of 10. The ABFM examinations that use this scale are built to common specifications according to the current ABFM certification examination blueprint. ¹⁰ Additionally, to facilitate direct comparisons, the difficulty of the questions and the estimated ability of the physicians are equated onto the same scale. Because an equated common scale was used, direct comparisons of scores are possible across exams and over time. In this study, the residents’ scores were their scaled scores on the ITE and FMCE.

ITE

The ITE is a formative low-stakes, multiple-choice question examination intended to provide residents with the opportunity to take a test with the same look and feel as the FMCE. During the study period, the ITE consisted of 200 to 240 questions. The ITE is administered by the residency program.

FMCE

Passing the FMCE is a requirement for earning ABFM certification; hence, it is a high-stakes examination. During the entire period of this study (2017-2022), the exam consisted of 260 scorable multiple-choice questions, and the minimum passing standard was a scaled score of 380. The FMCE is administered in a secured testing facility.

Table 1 displays the cohorts and examination calendar years for each cohort during residency. A cohort is defined as the year in which residents took their FMCE exam. Cohorts 2017-2019 are those that were prior to COVID-19, and none of their exams occurred during the COVID-19 pandemic. Cohorts 2020-2022 are those that had one to three exams impacted by COVID-19. The exam administrations that occurred under COVID-19 conditions are highlighted in Table 1. Specifically, cohort 2020 had only one exam (FMCE) impacted by COVID-19, whereas cohorts 2021 and 2022 had both ITE and FMCE exams impacted. This study period allowed for score trajectory analysis: PGY-3 for cohort 2021; PGY-2 and PGY-3 for cohort 2022.

As shown in Table 1, the impact of COVID-19 on exam performance should be manifested in the FMCE scores, specifically in the score trajectory during two growth periods: PGY-2 to PGY-3 and PGY-3 to FMCE. The growth period from PGY-1 to PGY-2 was excluded from this study because no cohorts were impacted by COVID-19 during that period. Because COVID-19 impacted cohorts at different exam instances, we conducted a multilevel regression analysis. We controlled for multiple exam scores from the same resident and residency programs clustering as random effects, and we controlled for residents’ demographic characteristics including gender, medical degree, country of medical school, race, and ethnicity as fixed effects. Our main exposure variable was the interval between examinations occurring during the COVID-19 2020-2022 cohorts (1 as yes; 0 as no). Specifically, we coded exam levels PGY-1, PGY-2, PGY-3, and FMCE as 1, 2, 3, 4 categorical levels in case the trajectory was not linear, and we created an interaction term between exam level and COVID-19 impacted exams. If the score growth during the two periods differed between COVID-19 impacted cohorts and prior to COVID-19 cohorts, we could conclude that COVID-19 had impact on the score growth, and the difference would inform us on the magnitude of the impact. Moreover, this coding scheme would also allow us to compare the FMCE score difference between the COVID-19 impacted cohorts and prior to COVID-19 cohorts. Finally, to control for the potential inflated Type I error caused by the large sample size in this study, we used effect size, denoted as η² , to measure the magnitude of score growth across multiple comparisons. Conventionally, an effect size of η² ≤0.01 is interpreted as a small effect; η² ≥0.06, but η² <0.14 as a medium effect, and η²≥0.14 as a large effect. ¹¹

Our analytic sample included 19,101 residents after excluding 3,724 residents with irregular progression through residency and an additional 12 residents with incomplete demographic information. The demographic information on the analytic sample is summarized in Table 2. The proportions of male and female physicians were about equal. Approximately 77% of the physicians held an MD degree, and the rest had obtained a DO degree. Two-thirds of the physicians were trained in US or Canadian medical schools and one-third in international medical schools. The proportions of race and ethnicity for each group were consistent with those found in other recent studies of residents. ⁹ Each cohort included approximately 3,000 examinees.

The exam performance of each cohort is depicted in Figure 1. The average scaled score (y-axis) of each exam (x-axis) is plotted for all cohorts. The dashed lines indicate the exam or exam trajectory that was impacted by COVID-19. The FMCE average scaled scores were similar across cohorts, but we found moderate declines in score growth during PGY-3 to FMCE for Cohort 2021 and almost no score growth during PGY-2 to PGY-3 for Cohort 2022. In multilevel regression controlling for demographic variables and clustering of scores within the resident and by program, we found that FMCE scores during COVID-19 decreased by only 6.6 points (P<.001, η² <0.01), representing only 1.2% of the average FMCE score prior to COVID-19. This comparison is similar to analysis of variance but takes demographic variables and clustering into account. The pass rates were almost identical for cohorts that experienced COVID-19 during residency and those that did not (99.8% vs 99.9%, χ² =0.0006, df=1, P=.980, η² <0.01). However, the learning loss earlier in residency was relatively substantive, as shown in Table 3. The increase in ITE scores from PGY-2 to PGY-3 during COVID-19 was 25.5 points lower than prior years, representing a 57.6% relative decrease in knowledge gain. This learning decline also was observed in the PGY-3 to FMCE growth period, but the magnitude was smaller (8.6 scaled score, representing 12.7% of the score growth in previous cohorts).

Table 4 further demonstrates the impact of resident characteristics. We found no statistically significant difference between female and male physicians’ FMCE scores, but physicians with an MD degree outperformed DO physicians. Similarly, US and Canadian medical school graduates scored higher on the FMCE than international medical school graduates. White physicians scored somewhat higher than physicians in the other groups. These results are consistent with previous literature. ⁹ Furthermore, the individual- and program-level clustering explained 47.8% and 8.6% of the total variance, indicating the necessity of using a multilevel model.

Our analysis of more than 19,000 graduating residents’ ABFM examination scores found nonsubstantive COVID-19 impact on FMCE scores but considerable declines in knowledge acquisition earlier in residency. This finding controlled for differences in resident characteristics and different learning environments of residencies, thus providing a more accurate and robust assessment of the impact of COVID-19 on residents’ medical knowledge acquisition. Overall, our analysis indicated that residencies were largely able to remediate knowledge deficits due to COVID-19 prior to residents taking the high-stakes FMCE. This remediation was possible because COVID-19 cohorts (2020-2022) had higher PGY-1 scores than pre-COVID-19 cohorts (2017-2019), as depicted in Figure 1. These elevated PGY-1 scores imply that the quality of their medical education remained largely unaffected, or at least less detrimentally impacted by the pandemic. Additionally, had they sustained a growth trajectory analogous to that of the pre-COVID-19 cohorts, they would have been expected to obtain higher FMCE scores. However, instead, they experienced two growth periods that were impacted by COVID-19: the transition from PGY-2 to PGY-3 for the 2022 cohort, and from PGY-3 to FMCE for both the 2021 and 2022 cohorts. For the 2022 cohort, residents’ scores showed almost no growth from PGY-2 to PGY-3, but their FMCE scores were compensated by higher PGY-1 scores and similar growth from PGY-3 to FMCE compared to pre-COVID-19 cohorts. For the 2021 cohort, residents’ higher or comparable PGY-1 to PGY-3 scores, compared to pre-COVID-19 cohorts, compensated for the impact on their score growth from PGY-3 to FMCE, resulting in final FMCE scores that were still comparable to those of pre-COVID-19 cohorts.

The comparability of the FMCE average scores with those of cohorts prior to the COVID-19 pandemic is noteworthy. The observed sustainability of FMCE scores has several plausible explanations. One explanation might be a potential delayed drop in the certification exam pass rate, which could be observed in the 2023 cohort that experienced their entire residency training during COVID-19. Second, the improvement in scores from PGY-3 to FMCE might be due to residency programs’ changing didactics in reaction to the drop in residents’ ITE PGY-3 scores. The use of a single FMC-Scale and the Bayesian score predictor enable programs to accurately gauge the probability of future scores based on ITE results. Third, when resources are scarce, residency programs and residents might deprioritize or have less motivation to perform well on the ITE (low stakes), yet they continue to prioritize the high-stakes certification exam even under limited resource conditions. This strategy is reasonable given the circumstances.

However, the ITE score drop is significant. The ITE score drop persisted even after controlling for resident characteristics, including prior scores, and clustering of residents in residencies with different clinical and learning environments. The relative learning drop percentage in the PGY-2 to PGY-3 period was more than 50%. In other words, this learning drop occurred during COVID-19 regardless of residents’ personal characteristics. Additional analysis further revealed that the learning drop occurred at a consistent rate across all race/ethnicity groups. The interaction between race/ethnicity and slope was not significant and is not shown in the main results. This consistency indicates that no differential effect learning drops were found with respect to race and ethnicity. This finding is consistent with our previous study, which showed that the learning trajectory is similar across different race and ethnicity groups during residency. ⁹

Indeed, a significant number of first-year residents, entering between 2020 and 2022, were not adequately prepared for clinical duties, primarily due to restricted training and experience during this period. ¹² This pattern has been noted consistently across various medical specialties throughout the nation. Program directors have consistently reported relatively low confidence in the readiness of new PGY-1 residents to assume specific tasks expected of them at the onset of their residency training. ¹³ This lack of experience may not be reflected in the final high-stakes certification exams, but may manifest in residents’ competency. The former focuses only on medical cognitive decision-making in applied settings, whereas the latter includes not only medical knowledge, but also skills and attitudes essential for providing high-quality care, such as communication skills, teamwork skills, system-based practice, and professionalism. In the long-term, the disruption of COVID-19 may serve as a reminder for the profession to continue implementing competency-based medical education. ^{14, 15} The COVID-19 pandemic is only one form of interruption. Given an interruption, if the goal is to provide a primary care workforce of similar competency, a redesign^{16, 17} of family medicine residency education to provide some additional learner-centered training might be necessary.

This study had several limitations. First, the results are applicable only to residents who went through residency training with regular progression. Although we could not compare scores for residents with irregular progression or missing scores, the percentage of regular versus irregular progression was relatively stable across cohorts (85%-90% vs 15%-10%). Second, medical knowledge is only one component of all the experience residents obtain from training, and other clinical experience was not measured in this study. Third, more learning metrics/tools need to be developed to measure residents’ learning and encourage learning behavior during residency. Measurement is crucial for the practical execution of competency-based medical education across all residency programs nationally in the future.

In summary, we found that disruptions in family medicine residency due to COVID-19 decreased the expected score growth more than half from PGY-2 to PGY-3, but the FMCE scores were not impacted substantively. While COVID-19 impacted learning, our findings indicate that residencies were largely able to remediate knowledge deficits before residents took the FMCE.