Evaluation of Family Medicine Residency Programs for Diversity, Equity, and Inclusion Milestones

With the increasing recognition of the long-term effects of racism on systemic health and health care inequities, medical education organizations have placed greater attention on diversity, equity, and inclusion (DEI). In 2020, the Accreditation Council for Graduate Medical Education (ACGME) released Common Program Requirements for residency programs to address DEI ¹ without providing strategies to measure programming or specific benchmarks. Additionally, while recent studies have documented underrepresented in medicine (URiM) resident and faculty recruitment efforts in various residency specialties ^2-10 and offered racial justice and health equity curricula, ^11-13 no large evaluations of DEI efforts in graduate medical education programs have been conducted. Smaller assessments have included the development of a climate assessment tool at a single program ¹⁴ or institution, ¹⁵ and the Racial Justice Report Cards developed by White Coats for Black Lives for medical schools and residencies, which were used to evaluate 26 medical institutions. ¹⁶ Another survey of family medicine program directors ¹⁷ asked questions about discrimination complaints in the training environment.

In 2022, the Association of American Medical Colleges (AAMC) released a report on DEI competencies across the learning continuum as a way to measure competencies of individuals, not programs. ¹⁸ Available DEI templates for business and nonprofit organizations ^19-22 do not meet the needs of graduate medical education, especially in terms of strategies to reduce bias in learner assessment, ^23-26 and faculty promotion. ²⁷ They also do not provide a framework to measure progress.

The Association of Family Medicine Residency Directors (AFMRD) Diversity and Health Equity (DHE) Task Force launched an initiative in 2019 to advance antiracism, diversity, and inclusion in family medicine residency training and developed a set of DEI milestones as the first known available tool to help residency programs identify areas of strength and opportunity within their programs. ²⁸ The DEI milestones provide a framework for programs to evaluate themselves across multiple DEI domains, including curriculum, resident and faculty recruitment, resident evaluation, and institutional learning environment. After creating these milestones using a process that mirrored the creation of the ACGME family medicine milestones, committee members piloted these milestones within their programs to evaluate for usefulness and understanding. The committee sought to document the status of DEI initiatives in family medicine residencies nationally by asking program directors to rate their progress on the milestones.²⁸ The findings establish a baseline that allows the AFMRD to tailor resource development to meet the needs of its members. The findings also provide individual programs with a benchmark to focus their activities and measure the effectiveness of initiatives undertaken to improve performance in the DEI domains, though individual programs may go about the evaluation process differently. This article describes the results of a baseline survey of US family medicine residency program directors implementing the DEI milestones.

We developed a cross-sectional survey and distributed it to program directors of ACGME-accredited family medicine residencies through the AFMRD member listserv. The survey was delivered online through the SurveyMonkey (SurveyMonkey Inc.) platform via a universal link; all responses were anonymous. Following an initial invitation to participate, an announcement was made at the annual meeting of the professional organization, and reminder emails were sent 2, 8, and 12 weeks later.

The survey instrument consisted of 12 quantitative items capturing residency program characteristics and the 5 DEI milestone ratings. The program characteristics questions mirrored those included in the Council of Academic Family Medicine Educational Research Alliance’s biannual survey, which is sent to program directors of family medicine residency programs to become familiar with them and to allow for future comparisons.

The curriculum milestone assesses implementation of DEI principles within the curricular content, such as the use of judgment-free terminology, social determinants of health (SDOH), and inclusion of antiracism content. The evaluation milestone assesses programs’ efforts to remove bias and increase objectivity within the promotion process along with inclusion of evaluation of residents’ ability to assess SDOH impact. The institutional milestone assesses the sponsoring institution’s integration with the community, hiring practices, and DEI policies and their application. The two pathways milestones for resident and faculty personnel, respectively, assess the programs’ recruitment, mentorship, and retention processes for residents and faculty. The full milestone framework has previously been described in detail. ²⁸

The AFMRD Diversity and Health Equity Committee, consisting of 11 members with extensive experience as residency program directors or associate program directors, convened and drafted the first iteration of the milestone framework. ²⁸ This framework was presented to more than 400 family medicine residency leaders for feedback. After refining items based on their input to improve readability and content validity, the survey launched on March 21, 2022, and closed on June 20, 2022.

We generated descriptive statistics, including item frequencies, percentages, and cross-tabulations, for all survey items. Analysis of variance (ANOVA) and independent samples t tests were used for subgroup comparisons. Statistical analysis was performed using SPSS Version 28 for Windows (IBM Corp.). A P value of <.05 was determined to be statistically significant.

The American Academy of Family Physicians (AAFP) Institutional Review Board (IRB) reviewed and approved this project (protocol #22-432) on February 26, 2022.

Of 588 eligible program directors, 194 (33%) completed the survey. Respondents represented 48 states and US territories; additional characteristics of respondents’ residency programs are reported in Table 1.

We calculated mean ratings (range 0.5–5.0) for all five DEI milestones (Figure 1). Overall, the curriculum milestone was rated the most advanced along the scoring rubric (mean=2.54, SD= 1.03), whereas the faculty pathway (mean=1.94, SD=1.04) and resident pathway (mean=2.02, SD=1.06) milestones were rated less advanced. The faculty and resident pathway milestones demonstrated the highest inter-item correlation (r=0.81, P<.01).

On average, the 100 programs with 25% or higher URiM residents reported significantly higher ratings on the curriculum (mean=2.71, SD=1.04, t[192]=–2.45, P=.02), faculty (mean=2.19, SD=1.02, t[192]=–3.51, P<.01), and resident (mean=2.26, SD=1.12, t[192]=–3.40, P<.01) pathway milestones compared to their counterparts with fewer URiM residents, as reported in Table 2.

We performed one-way ANOVA to compare average milestone ratings among residency programs of different complements. We observed significant differences for the curriculum (F[2,191]=3.95, P=.02) and faculty pathway milestones (F[2,191]=3.70, P=.03). Tukey’s HSD test for multiple comparisons revealed that larger programs (with greater than 31 residents) were rated significantly higher on the curriculum milestone than smaller programs with fewer than 19 residents (d=0.57, CI=0.09–1.06, P=.02). Regarding the faculty pathway milestone, larger programs rated themselves higher, on average, compared to smaller programs (d=.57, CI=0.07–1.06, P=.02), as reported in Table 3.

Similarly, one-way ANOVA revealed significant differences among residency programs of different institutional affiliations. On average, university-based programs reported higher milestone ratings compared to their community-based counterparts on all but the resident evaluation milestone, as denoted in Table 4.

This DEI milestone survey of US family medicine program directors found that programs that were university-based, located in larger communities, and had a large number of residents scored themselves higher on most but not all milestones when compared to their counterparts. Additionally, programs with 25% or higher URiM residents reported significantly higher scores on curriculum, faculty pathway, and resident pathway milestones. Programs overall rated themselves highest on the curriculum milestone.

Programs located in larger communities and those that were university-based scored higher on milestones. This may be because they drew a more diverse resident and faculty base that was more focused on DEI. Also possible is that institutional DEI initiatives at the university level provided greater access to DEI resources, thereby advancing DEI efforts.

We found higher scores in the curriculum, faculty pathway, and resident pathway milestones in programs with more URiM residents. Residents who identified as URiM could have been drawn to these programs, and programs might have been motivated to prioritize issues encompassed in these milestones to better meet the needs of their URiM residents who might enhance the program’s demographic diversity.

The scores for the curriculum milestone were higher than scores for other milestones. One explanation is that curriculum is more easily modified—with the effect of changes seen more quickly—than more complex domains like pathway and institutional domains, especially in larger institutions. In addition, resident feedback might have a more immediate and actionable effect on curriculum as opposed to the other domains.

The response rate was 33% of surveyed programs, and some programs may have program directors who are not represented in the AFMRD membership. Program characteristics are self-reported, and categories were not defined; however, these categories are used in biannual Council of Academic Family Medicine surveys and are familiar to program directors. Additionally, no public source of program characteristic data exists to determine whether this is a representative sample. Possibly, program directors from programs more attuned to DEI issues were more likely to complete the survey, which might have either over-sampled higher scoring programs or conversely over-sampled programs that rated themselves more critically. Milestone scores were self-rated, and programs with similar processes in place may have rated themselves differently. Because our survey is meant to be a tool that allows programs to rate themselves over time, the likelihood is that programs will continue to rate themselves in the same way, contributing to the validity of the rating rubric within each program. Also, a desirability bias may have impacted the ratings despite the anonymity of the survey. A survey completed by raters external to the program could overcome this potential bias.

The findings of this study provide a baseline assessment of the status of DEI in family medicine residencies. Next steps include a follow-up study of family medicine residencies at 1-year or every-other-year intervals to track changes in milestone scores over time. A mixed methods approach that includes qualitative surveys of programs to see how they are using the milestones and what interventions they have undertaken also would be useful. This information could include where the milestones are being discussed, such as among leadership only, with the program evaluation committee, or with all program members. Assessment of milestone progress also will allow organizations such as the AFMRD and the Society of Teachers of Family Medicine to focus on areas of demonstrated need, such as faculty development. Performing DEI milestone studies across other specialties and studies to assess reliability of the milestones among programs will be important as well.

Programs scored highest in the curriculum milestone and lowest on the faculty and resident pathway milestones. Larger and university-based programs as well as those in larger cities scored higher on several milestones. Milestone data may be useful in supporting residency programs as they assess their institution’s developmental progress across key DEI domains.

Information from this article has been presented at or accepted to be presented at the ACGME Equity Matters Symposium, the ACGME Annual Education Conference, the AAFP Resident Leadership Summit 2023, and the 2023 Society of Teachers of Family Medicine Annual Meeting.

The application fee for the AAFP IRB was paid for by the AFMRD.