The Impact of Virtual Interviews on Recruitment and Implicit Bias

Virtual interviews (VIs) have theoretical benefits to both programs and applicants, including reduced cost, time, and increased geographical diversity.1-3 In contrast, benefits of in-person interviews for applicants include enhanced understanding of residency culture through interactions with current personnel, informal observation of resident, faculty, and staff, and direct exposure to the locale and working environment.4-7 A unique, unquantified outcome of VIs is the potential for bias to adversely affect recruitment of minority applicants underrepresented in medicine (URM).8 The use of VIs present an opportunity to reduce bias, but also open the door to new risks. The fact that VIs invite interviewers into applicants’ personal space opens potential for bias, ranging from an applicant’s artwork and home décor to technical competence and internet quality.9 A previous pilot in emergency medicine used the Standardized Virtual Interview (SVI) tool, which was found to be a reliable and valid assessment of interpersonal and communication skills, as well as professionalism. These VIs did not disadvantage applicants and the holistic screening process to select applicants who interviewed virtually created a more diverse applicant pool than the previously used in-person processes. Unfortunately, the pilot was stopped due to lack of evidence to support SVI, uncertainty around costs, and unfavorable student perceptions on its validity.10, 11

As part of the effort to limit the spread of COVID-19 in fall 2020, the Association of American Medical Colleges (AAMC) and Accreditation Council for Graduate Medical Education (ACGME) strongly encouraged all residency programs to conduct interviews virtually, either by phone or video conferencing.12, 13 Previously, most residency programs did not routinely offer VIs. This new undertaking was implemented by residencies with no established standard guidelines; therefore, the AAMC and ACGME published resources.14 Thoughtful planning of the interview experience was advised to ensure that all applicants have a fair and standardized experience,15, 16 including interviewer training about unconscious bias.17 Practical suggestions included portraying the culture of the program and city, accurately depicting program information, and offering virtual interactions to increase exposure to residency program culture (eg, virtual social events with current residents, updating the program webpage to highlight the locale, and improving program website accuracy and maneuverability).18-22

Currently-published information addresses neither residency program director (PD) adoption of the 2020 AAMC and ACGME recommendation for VIs, nor their satisfaction with match results. We performed a survey of family medicine (FM) PDs about their interviews for the 2020-2021 academic year (AY21) to investigate these research questions:

1. What percent of PDs enacted the VI recommendations?
2. What was the impact of VIs in AY21 on PDs’ general perceptions of the recruitment and match process?
3. How did PD’s general perceptions of bias in VIs change as a result of the AY21 interview season and how did this impact PDs’ plans for the future?

The survey questions were part of a larger omnibus survey conducted by the Council of Academic Famil Medicine Educational Research Alliance (CERA). The methodology of the CERA Program Director (PD) Survey has previously been described in detail.23 The CERA Steering Committee evaluated questions for consistency with the overall subproject aim, readability, and existing evidence of reliability and validity. The committee pretested the survey on FM educators who were not part of the target population, modifying questions for flow, timing, and readability in response to feedback. The American Academy of Family Physicians Institutional Review Board approved the project in April 2021, and data were collected from April 14, 2021 to May 17, 2021.

The sampling frame for the survey was all ACGME-accredited US family medicine residency PDs as identified by the Association of FM Residency Directors (AFMRD). Email invitations to participate were delivered with the survey utilizing the online program SurveyMonkey. Four follow-up emails to encourage nonrespondents to participate were sent weekly after the initial email invitation and a fifth reminder was sent 2 days before the survey closed. There were 699 PDs at the time of the survey; eight had previously opted out of Survey Monkey surveys. The survey was emailed to 691 individuals. Of the emails sent, 35 bounced and one additional PD opted out of SurveyMonkey surveys, leaving 655 invitations delivered. The survey contained a qualifying question to remove programs that had not had three resident classes; 36 PDs indicated that they did not meet criteria. These responses were removed from the sample, reducing the sample size to 619 and respondents to 287.

The full survey questions are available as a digital appendix to this article (https://journals.stfm.org/media/5113/appendix-keister.pdf). In summary, we asked PDs who conducted VIs about their general perceptions regarding administration, diversity, bias, the ability to communicate program culture and to assess applicant alignment with program values. We also queried their satisfaction with the results of the 2021 National Residency Matching Program (match) and about plans for interview type and structure when restrictions are lifted. Response options to the survey questions included yes/no, multiple choice, and a Likert scale (strongly disagree, disagree, agree, strongly agree). In several cases, the Likert-scale responses were collapsed for analysis (eg, agree and strongly agree were combined to be equivalent to “yes”).

We assessed bivariate associations between variables using Fisher exact tests. We used simple and multiple logistic regression to further assess associations between match satisfaction and minority representation and cultural communication and alignment variables. To account for multiple testing bias, we used the Benjamini-Hochberg procedure to control the false discovery rate at α=0.05 in all analyses pertaining to minority representation or culture communication. We calculated adjusted P values using the p.adjust procedure from the statistical package in R. We conducted all analyses in R software v4.0.5 .

The overall response rate was 46.4% (287 of 619 surveyed). The surveyed PDs were of equal gender distribution and the majority were White or non-Hispanic/Latino. Most residencies reported that the percentage of URM in medicine among their residents and faculty was less than 20% (Table 1). Most residency programs represented in this study were community based, university affiliated, and were distributed throughout the United States (Table 1). The data analyzed for this study have been deposited in the CERA Data Clearinghouse and are available at https://www.stfm.org/publicationsresearch/cera/pasttopicsanddata/pasttopiclist/#tab-6378.

Recruiting seasons prior to the July 2020 - June 2021 academic year (AY21) were largely conducted in person. During AY21, 93.1% (n=244) of programs conducted their interviews virtually. At the time of the survey, most programs planned to offer both virtual and in-person interviews with an in-person second look when in-person interviews are allowed (Table 2). Most program directors (n=238, 83.0%) reported their general perception that they had experienced a decrease in costs for AY21. However, for most PDs (n=247, 86.0%) cost was not a major factor in their decision to offer VIs in the future. More than half of the surveyed programs experienced an increase (n=153, 53.1%) in applicants for AY21 compared to prior years, while 43.89% had no increase in applicants. PDs did not feel that the AY21 interview season decreased their administrative burden (n=193, 67.2%). Neither administrative burden (P=.059) nor cost (P=.844) were a factor in the plan to offer only VIs in the future. 

Most PDs that conducted VIs in the AY21 season were satisfied with their match results (n=220, 83.9%) with no difference between those that conducted all VIs and those that did not (OR 1.2, P=.99). There was no difference in match satisfaction between those that plan to offer VIs in the future and those that do not (OR 2.51, P=0.088). Programs that were able to involve residents to the same degree as prior years were more satisfied with their match results (Table 3). 

Most PDs were able to involve residents and to communicate their program’s culture, values, and mission to the same degree as prior years (Table 2). Compared to VIs, PDs indicated in-person interviews allowed for a more comprehensive understanding of the applicants’ alignment with the program, and the limited ability to observe nonverbal behaviors of applicants in VIs reduced the accuracy of their applicant assessments. The impact of VIs on communication or assessment of applicant alignment with program values did not correlate with PD plans to make changes in the future to the structure of the VI season (Table 4). PDs who were able to communicate values or involve residents were more satisfied with match results. If they planned to make major changes to help applicants understand the program, they were less satisfied with match results. If they felt that in-person is better than VIs for assessing cultural fit or they could not read verbal cues in VIs, they were less satisfied with match results. With an additional adjustment for program size, program type, percentage of international medical graduates, percentage of minority residents, percentage of minority faculty, years of experience as PD, PD gender, concern about bias towards minorities, PD ethnicity, and PD race, these two associations were no longer significant, which indicates that the differences in satisfaction may have been due to program factors rather than an actual PD preference. The match satisfaction for PDs who were able to communicate values and involve residents persisted despite this adjustment to exclude program factors (Table 3).

Of the programs that conducted VIs, most PDs were not concerned that VIs would magnify implicit bias against underrepresented minorities either prior to or after the AY21 season (Table 2). PDs who were concerned about VIs increasing bias against minorities before moving to VIs were still concerned about the existence of bias against minorities after the pandemic (OR 8.81, P<.001). To reduce implicit bias, most PDs (n=140, 48.9%) plan to have general training for the entire program prior to the next season. Those who were concerned about VIs contributing to bias before AY 21 were more likely to take action to reduce bias in the next recruiting season (OR=3.29, P=.02), while those who were still concerned after AY21 were not more likely to take action to reduce bias for the next interview season (OR 1.24, P=.73). PDs who were concerned that VIs increase bias aft
er AY21 were less likely to be satisfied with match results (Table 3). Many programs (n=229, 79.9%) noted an increase in geographic diversity of their recruiting pool; this increase did not correlate with improved match satisfaction (OR 0.99, P=.98).

This anonymous survey of family medicine residency program directors addresses the call of the Coalition for Physician Accountability,24 the Council for Academic Family Medicine (CAFM)25 and others7,22 to study the impact of VIs on stakeholders in the residency interview process. The vast majority of programs complied with recommendations to perform only VIs in AY21. Despite the sudden switch from in-person interviews for most programs, more than 80% of PDs were satisfied with their match results at the time of the survey. However, only 4% of PDs intended to remain entirely virtual when in-person interviews are permitted, indicating that few PDs were fully satisfied with the entirely virtual process. 

Concerns that VIs detract from the ability of candidates to evaluate the values and culture of a residency and to interact with current residents have been cited.5,22 In this survey, PDs who felt that they had communicated their values well were more likely to be satisfied with their match, as were those who were able to include residents in their VIs to the same degree as previous years. Thus, intentional efforts to clearly communicate values about the culture of the residency and to include residents in VIs are best practices to enhance match satisfaction for residency programs. 

Although most residency directors reported a perception of a reduction in cost to the program in AY21, the reduction was not a predictor of their likelihood to continue offering only VIs in the future. In addition, there was no perceived change in administrative burden and no correlation of increased geographic diversity of applicants with match satisfaction. These three factors have been reported as major advantages of VIs1,22 for residencies. Given that none of these was a perceived benefit to PDs, it is not surprising that most PDs preferred to include in-person interviews in future recruiting. Notably, 1 year may not be enough time to recognize the benefits of VIs. It is possible that the time required and administrative burden for interviews would be less after VIs are in place for several years. Additionally, PDs might become more appreciative of the reduced costs if they were able to reallocate those funds and witness the benefits. 

FM PDs’ perspectives on bias against URMs in VIs are interesting. Few were concerned about VIs increasing bias prior to AY21, and that number decreased slightly after VIs were implemented. However, this apprehension was a driver of match dissatisfaction. Interestingly, more PDs intend to conduct training about implicit bias prior to the next recruiting season than were worried that VIs increased implicit bias. This indicates that even those who believed that VIs did not compound implicit bias were still concerned about bias in their interviewing process. Although training interviewers about implicit bias is an important intervention, this training alone is unlikely to reduce the impact of bias on selection.26 Additional suggested interventions include removing photos from applications when choosing interview candidates, removing academic metrics (eg, USMLE scores), and reviewing implicit bias reduction techniques (eg, implicit bias reduction cheat sheet) just before conducting interviews.27 We find it surprising that PDs who were concerned about VIs’ impact on bias before and after AY21 did not report more likelihood to future action to reduce bias. It is possible that these PDs had already acted to address the risk of bias. In addition, these PDs may have been planning to shift back to in-person interviews for the next recruiting season. 

The survey has several limitations that may impact the generalizability of its findings. First, the survey went only to family medicine residency directors. PDs from other disciplines may have experienced VIs differently. Also, we surveyed PDs’ general perceptions from the interview season and did not ask for specific responses. For example, we asked whether the interview season had cost more, the same or less rather than asking for specific dollar amounts.

The response rate of the survey was 46%. It is possible that the results do not reflect the overall population of PDs, although the demographics of respondents were similar to the published demographics for current FM PDs.28 Our sample did have a lower percentage of Asian respondents (1% compared to 7% nationally), a higher percentage of Black respondents (9% vs 5% nationally), a higher percentage of White respondents (83% vs 71% nationally), and a smaller percentage of respondents who do not respond/chose not to disclose (2% vs 15% nationally). These demographic differences could have caused differences in our respondents’ perceptions of the impact of VIs, especially regarding concerns about bias. It is also important to mention that only the PDs’ perceptions were surveyed. We did not assess whether PDs understand how bias affects other interviewers or applicants, nor did we assess applicants’ perceptions of the impact of VIs on bias. The fact that the PDs were largely Caucasian could have led to a gap between their perceptions of bias and those of the applicants or other interview participants.

In addition, because the survey was distributed before AY21 applicants matriculated, PD match satisfaction may not reflect residents’ actual performance. We also must highlight that this survey occurred prior to the AAMC and CAFM recommending that programs “strongly consider” continuing VIs for AY22.14,24,25 Therefore, we are unsure how to interpret the responses about future plans in light of these recommendations, as the phrases “strongly consider” from CAFM and “permitted” in the survey may be interpreted differently by PDs. Also, the breadth of the issues around bias during residency interviews were more widely disseminated as a part of recommendations (AAFP) shared after the survey period. For example, the inequitable impact of travel costs for in-person interviews to those from less affluent backgrounds,3 the possible favoritism that could be shown toward candidates who interviewed in-person during a hybrid interview processes,2,29 and the worry that second looks occurring before rank list submission might bias programs’ rank decisions were highlighted in the recommendations.25 Although some PDs were aware of these concerns prior to the recommendations, the issues may not have been evident to all PDs at the time of the survey. In addition, we asked specifically about racial bias in VIs. These other issues reveal possible economic and social biases that this survey did not address. It is possible that the survey results, especially the intent of the majority of PDs to offer both in-person and Vis, would be different as more PDs have become aware of these concerns. 

Conversations among PDs since the release of the recommendations to continue virtual-only interviews25,30 reflect the possibility of additional bias and unintended consequences. For example, some have postulated that community-based residencies may be more impacted than university-based programs by the inability to have applicants in person for interviews, because medical schools are generally in university-based environments and students may be more likely to choose what they know. In addition, little attention has been given to the bias created by audition rotations in which students rotate at a residency site. Clearly, students who perform these rotations have a much better chance to assess the programs, and vice versa. However, the processes used to select which students perform these rotations and the impact on rank list decisions are not well studied. 

Advancing the understanding of the medical education community’s implementation of VIs remains of paramount importance. Robust investigation of the perceptions of applicants, PDs from other disciplines, and faculty members about VIs is critical. More information about the specific technologies used for VIs (eg, telephone vs video conference, specific video vendors) and whether these impact PD and/or applicant perceptions of VIs would be valuable. The impact of efforts to reduce bias in all interviews, including virtual, must be further pursued, as must further understanding of residency PDs’ perceptions about the impact of VIs on student cost, racial bias, economic bias, social bias, and other factors. 

The call for rapid implementation of VIs due to the COVID-19 pandemic was largely executed by FM residency programs, and most PDs were satisfied with the match. The ability to include residents and to convey a residency culture and values during the VI process are associated with higher match satisfaction. A better understanding of the impact of implicit bias in residency VIs and the interventions to mitigate that bias are important steps as national organizations consider the future of interviews for residency positions.