Holistic Review in Family Medicine Residency Programs: A CERA Study

Racial and ethnic diversity in the physician workforce, particularly a higher number of physicians underrepresented in medicine (URiM), improves patient care access and quality of care. ^{1, 2} URiM is an Association of American Medical Colleges (AAMC) designation for groups that are underrepresented in medicine compared to their representation in the general US population: Black/African American, Hispanic/Latino/of Spanish origin, Native American/Alaskan Native/Indigenous, and Native Hawaiian/other Pacific Islanders. ³ URiM physicians are more likely to practice in primary care and work in underresourced communities. ^{4, 5} Racial concordance between patients and physicians also contributes to a more effective therapeutic relationship, improved patient satisfaction, and improved patient outcomes. ^{2, 6, 7}

Unfortunately, this underrepresentation has been found among American Board of Family Medicine certification candidates as recently as the years 2010 to 2020, with the percentage of Black/African American individuals at 8.1% compared to 12.8% of the US population and the percentage of Hispanic/Latino/of Spanish origin individuals at 9.3% compared to 18.4%. ⁸, ⁹ The medical residency selection process generally is divided into three primary stages: initial screening of applicants for interviews, interviewing applicants, and ranking applicants. Historically, residency programs have used academic metrics such as United States Medical Licensing Examination (USMLE) scores when evaluating applicants and often screen out applicants based on those scores early in the process. ¹⁰ However, previous research has shown that USMLE Step 1 and 2 scores do not correlate with acquisition of clinical skills ¹¹ and may limit recruitment of a diverse physician workforce, because URiM and students who identify as women historically have performed lower than White male counterparts on standardized exams and are therefore negatively impacted by such an evaluation system. ¹²

To improve the diversity of the physician workforce, AAMC has long championed holistic review in the admissions process at the level of undergraduate medical education. ¹³ Holistic review is defined as

In practice, a program instituting holistic review determines for itself a rubric, including applicants’ experiences, attributes, and academic metrics, necessary to achieve its specific mission. Programs may look into not only traditional measures of diversity such as race, ethnicity, and gender, but also factors like socioeconomic status, language capabilities, rural representation, and distance traveled ¹⁵ (ie, how far a student has come in light of discrimination or a lack of resources or support) to determine how equipped an applicant might be to meet the needs of diverse patient populations. Since the introduction of holistic review, undergraduate medical education admissions have been successful in having more diverse interview pools than when using academic metrics alone. ¹⁶

Implementation of holistic review at the graduate medical education level has the potential to improve the diversity of the physician workforce. However, in the aftermath of the Supreme Court decision Students for Fair Admissions, Inc. v President and Fellows of Harvard College ¹⁷ to end affirmative action in college admissions, many residency programs are seeking methods to assess an applicant’s diverse lived experiences. At a large academic center trying to achieve improved racial concordance, using a systematic holistic application review process significantly increased the proportion of interviewed and matched candidates who identified as URiM. ¹⁸

Currently, no data has been published on the prevalence of holistic review in the family medicine residency program (FMRP) selection process. Additionally, while FMRPs may take into consideration components outside of academic metrics, whether each component of the experiences, attributes, and academic metrics framework is weighed equally or whether programs still filter out candidates based solely on academic metrics is unclear. To explore the current scope of holistic review within FMRPs, this study developed a series of questions that were included in the 2023 family medicine residency program directors’ survey sent by the Council of Academic Family Medicine Educational Research Alliance (CERA). Survey objectives were to determine: (a) knowledge, skills, and attitudes of holistic review; (b) current prevalence and intensity of formal holistic review in FMRPs; (c) barriers to implementation of holistic review as experienced by FMRPs; and (d) program characteristics’ correlations with the presence of holistic review. 

The survey questions were part of a larger omnibus survey conducted by CERA. The larger survey included items assessing demographic and program characteristics that were available to all study teams contributing survey items. Program characteristics and program director (PD) demographic variables used in the study included measures of program type, size of the service community, number of residents, years as a PD, and PD self-identification as URiM. We mapped existing omnibus categories of service community size to Health Resources and Services Administration definitions of rurality by creating a variable for rural as communities of less than 30,000 and urban as greater than or equal to 30,000. ¹⁹

The study team developed 10 items that were added to the omnibus survey related to holistic review and additional program characteristics. Survey questions resulted from a literature review. For this survey, we used the AAMC definition of holistic review, shared earlier; that definition was provided to respondents to provide a uniform understanding prior to answering the developed items. Table 1 describes the items developed for the survey.

The CERA Steering Committee evaluated questions for consistency with the overall subproject aim, readability, and existing evidence of reliability and validity. Pretesting was done on family medicine educators who were not part of the target population. Questions were modified for flow, timing, and readability. The project was approved by the American Academy of Family Physicians (AAFP) Institutional Review Board. Data were collected between April and May of 2023.

The survey was sent to all 745 US family medicine residency PDs as identified by the Association of Family Medicine Residency Directors. Email invitations to participate were delivered with the survey, available through the online program SurveyMonkey (SurveyMonkey, Inc). Three follow-up emails to encourage nonrespondents to participate were sent weekly after the initial email invitation, and a fourth reminder was sent 1 day before the survey closed. Six email addresses were undeliverable, leaving 739 invitations delivered. The survey contained a qualifying question to remove programs that had not had three resident classes. Forty-eight program directors indicated that they did not meet the criteria; these responses were removed from the sample, reducing the sample size to 691.

We used frequencies for univariate descriptions. We did bivariate comparisons using ꭓ² comparisons for categorical variables. We performed Kendall τb correlations for ordinal variables to account for smaller datasets and the presence of ties among the data. ²⁰, ²¹ We used a level of significance of P<.05. Analyses were conducted using Stata 17 (StataCorp LLC).

The overall response rate for the survey was 44.72% (309/691). Most of the FMRPs were community-based and university-affiliated (58.8%), 24.1% were community-based only, and 16.2% were university-based. Only 11% of FMRPs were considered rural, serving a community size of less than 30,000. The rest of the FMRPs were more urban, with 20.3% of them serving a community size of 75,000 to 149,999, 25.1% serving a community size of 150,000 to 499,999, and 17.2% serving a population greater than 1,000,000. In terms of the number of residents within FMRPs, 40.2% of them had less than 19 residents, 44.3% had 19 to 31 residents, and 15.1% had more than 31 residents. The distribution in the duration of years that PDs had been in their role was approximately even: 26.8% were PDs for less than 3 years, 28.2% for 3 to 5 years, 18.6% for 6 to 9 years, and 26.1% for at least 10 years. Only 18.9% of the PDs reported their own status as URiM.

In terms of knowledge, skills, and attitudes, 75% of FMRPs agreed/strongly agreed (38% agreed; 37% strongly agreed) with the use of a holistic review approach for their selection process. Also, 72% of FMRPs self-reported understanding holistic review, with 50% self-reporting understanding to a large extent and 22% self-reporting understanding it completely. The majority of FMRPs also reported using holistic review to a large extent in all of the selection process stages: screening, interviewing, and ranking (Figure 1).

The top three barriers to implementing a holistic review framework were the number of applicants, the increased resources associated with it, and lack of consensus within programs on the holistic review approach (Figure 2).

We found a significant negative correlation between the length of time served as PD and agreeing with holistic review as a selection process (Table 2). Similarly, we noted significant negative correlations between the length of time served as PD and use of holistic review in the screening, interviewing, and ranking stages of the selection process. In contrast, we identified significant positive correlations with aspects of holistic review with larger numbers of residents (ie, screening); higher rates of URiM residents (ie, interviewing); higher rates of URiM faculty (ie, self-reported understanding, agreement, screening, interviewing, and ranking); higher rates of URiM patients (ie, agreement, interviewing, and ranking); and larger community sizes (ie, agreement). We also found consistently significant positive correlations between FMRPs’ agreement and self-reported understanding and implementation of holistic review throughout the three stages of the selection process (Table 3).

The data suggest that the majority of FMRPs not only perceive the use of holistic review favorably in the selection process for residents but also currently implement it throughout the process. These programs also were found to have higher rates of URiM patients, residents, and faculty compared to programs that did not prefer holistic review.

These findings are reflective of a scoping review that found that US residencies and fellowships aiming to increase their diversity, especially their proportion of URiM makeup, are using holistic review as one of their strategies. ²² Specific programs, such as the University of Texas Health Science Center at Houston Internal Medicine Residency Program ²³ and the Boston Medical Center Family Medicine Residency Program, ²⁴ have reported increased diversity after their own implementation of holistic review.

To our surprise, newer PDs reported using holistic review to a greater extent than more experienced PDs. This might be due to holistic review being a more recent approach to recruitment compared to the traditional recruitment process that emphasized academic metrics. Given that newer programs were disqualified from the survey, the use of holistic review possibly was underestimated if those programs are creating selection criteria during a time when holistic review is becoming more utilized.

Our study had several limitations. First, given the nature of inquiry, some social desirability bias in the responses is likely. For instance, the survey called for self-assessments of understanding holistic review rather than an objective measure of understanding—potentially overestimating how well PDs understand it. Similarly, PDs self-reported at what stages holistic review was used, but the study did not assess how it was applied. Given the limited number of survey questions, additional studies are needed to assess how programs apply holistic review to each process of the application cycle. Second, while consistent with many national surveys, the overall survey response rate was only 44%, which may represent some bias in sampling and limit the generalizability of the findings. Third, the structure of CERA surveys allows only one answer per question and does not allow “all applicable” responses, limiting nuance in the response options provided by respondents. Furthermore, no free text option was offered for those who answered “Other” to barriers of holistic review implementation, limiting our ability to explore other potentially important program aspects. We note that our survey item designed to elicit respondents’ level of agreement with using a holistic review approach included the word “agree” in the question stem and had five levels of ordinal agreement across which respondents could differentiate levels of agreement; this wording may have been confusing to some respondents. However, the item appeared to function appropriately because we had a good distribution across response options. Because causality and directionality are not possible with cross-sectional data, this study’s conclusions can be limited only to associations. More elaborate study designs would be necessary to assess the direction of associations, including higher endorsement and use of holistic review in larger programs/communities and programmatic/community diversity resulting from the existing diversity or the driving of diversity from the result of the use of holistic review.

On the other hand, this study was the first to demonstrate just how prevalent and favorable the majority of FMRPs are to holistic review. However, the increased number of applicants and increased resources needed to implement holistic review are important barriers that FMRPs need to tackle. The mapping of experiences, attributes, and academic metrics to mission values also may be a new experience for some FMRPs. As a result, disseminating models and interdepartmental and system (eg, AAMC, Association of Departments of Family Medicine, Society of Teachers of Family Medicine, AAFP) mentorship would be beneficial. Exploring partnerships and sharing of resources across institutions already implementing holistic review, participating in workshops, ²⁵ and using mission-based filters in the Electronic Residency Application Service (ERAS) ²⁶ are some opportunities that could help overcome these barriers. With the 2023 ERAS application updates, geographic preferences and impactful experiences also can potentially help programs implement holistic review.

Future work should examine the specific scoring rubrics of FMRPs that use holistic review to assess how it is being implemented. Additionally, quantifying the increased cost of holistic review would be helpful, given increasingly tight hospital budgets and pressure on residencies to cut costs. Cost could also be a metric to increase accountability in future graduate medical education payment reform. While this study examined the associations of holistic review with programs’ racial and ethnic diversity, other diversity factors that are historically minoritized in medicine—such as gender identity, sexual orientation, socioeconomic status, rural status, resilience, and distance traveled—should also be evaluated.

 In the aftermath of the Supreme Court decision to end affirmative action in college admissions, ¹⁷ many residency programs are seeking methods to assess an applicant’s diverse lived experiences. The holistic review approach, which considers experiences and attributes in addition to academic metrics, can incorporate an applicant’s lived experiences of resilience and distance traveled to invite a diverse pool of applicants for interview selection, independent of race and ethnicity. Increasing the diversity of the physician workforce has been recognized as having an important role in addressing the large racial and ethnic health outcomes disparities in the United States. Our study revealed that holistic review has increasingly become a purposeful recruitment tool for FMRPs to enhance their diversity makeup, particularly in groups that are considered URiM. Use of holistic review has the potential to drive more URiM representation into the physician workforce. ¹⁸ Ultimately, this effort has the potential to significantly improve the health and satisfaction of groups that have been historically marginalized and the US patient population as a whole.