Evaluating Changes in Family Medicine Applicant Characteristics Following the Onset of Virtual Interviewing

Interview days have always been an opportunity both for medical students applying for residency programs as well as residency programs courting applicants to obtain a more personal impression of each other.1 The interview experience allows programs to highlight their unique strengths, while simultaneously assessing applicants’ interest in and fit with the program, as well as communication skills, depth of understanding of the specialty, and maturity.1 It also provides applicants an opportunity to interact with program faculty and residents, in order to assess the overall culture and camaraderie, thereby determining fit from their perspective.2 Traditionally, interviews have occurred in person. However, in 2020 safety concerns, social distancing requirements, and global travel restrictions related to the COVID-19 pandemic prompted an abrupt shift to virtual interviewing for graduate medical education (GME) residency applicants.3

A joint statement in 2021 from the American Academy of Family Physicians, Society of Teachers of Family Medicine, Association of Departments of Family Medicine, North American Primary Care Research Group, and Association of Family Medicine Residency Directors, hailed virtual interviews as “an equitable option for candidates as it eliminates the expense of travel” and hypothesized that “removing financial barriers may increase the diversity of candidates for programs.”4 Although virtual interviewing had been explored prior to the pandemic, trials resulted in mixed successes. Multiple prepandemic studies in family medicine and surgical residencies comparing virtual to in-person interviews found virtual interviews to be time and cost effective.1,5 In contrast, a 2019 investigation of medical student and resident perceptions indicated a preference for in-person interviews in both groups.6 In addition, a structured video interview pilot in emergency medicine was abandoned due to lack of interest, cost, and student perceptions.7 However, further studies leading to best practices did not emerge in time for the 2020 interview season, and residency programs launched into virtual interviewing with little experience or guidance.

In the wake of the first cycle of virtual interviews, multiple specialties including obstetrics and gynecology fellowships, orthopedic surgery, and internal medicine subspecialty fellowships published evaluations of their virtual interviewing experiences, which again demonstrated cost savings to programs and applicants, and overall acceptability of the process.8-14 However, changes to the diversity of applicants were either not evaluated, or evaluated and found to not be affected by transition to the virtual platform.15 Furthermore, information from family medicine programs is lacking.

Our multisite study sought to add to the understanding of the impact of virtual residency interviews in family medicine, including the impact on applicant and interview process characteristics. In addition, this study sought to identify changes to the geographic diversity of family medicine applicants following the implementation of virtual interviewing as well as the impact on applicants with backgrounds historically underrepresented in medicine (URM). Finally, this study explored changes in the overall number of applicants given the concerns related to the ratio of applications to number of available residency positions.

Five family medicine residency programs, which were selected via convenience sample, participated in this study. Programs ranged in size from 12 to 82 residents, included urban and suburban settings, consisted of university based, community based/university affiliated, and community based/unaffiliated settings, and included a variety of curricular emphases (eg, inpatient, outpatient, obstetrics, rural). The following institutions from various geographic regions submitted data:  Waukesha Family Medicine Residency at ProHealth Care in Wisconsin, St Marks Family Medicine Residency in Utah, University of Toledo Family Medicine Residency in Ohio, University of Mississippi Medical Center in Mississippi, and JPS Health Network in Texas. The in-person interview years prior to COVID-19 included the recruitment cycles 2017/2018 through 2019/2020. The two virtual interview years during COVID-19 included the recruitment cycles 2020/2021 and 2021/2022. Of note, one program had a hybrid interview option in 2021/2022. In addition, one program had missing data for the “Offered to Interview” variable for in-person years, so we used “Interviewed” as a proxy.

We extracted data from the Electronic Residency Application Services (ERAS) individually by program, de-identified them, and then transmitted them to the study’s data manager for analysis. From the ERAS system, extracted information included the permanent zip code of applicants, the gender of applicants, applicant racial/ethnic self-identity, the country of the applicants’ medical school (international vs United States), and if they were offered to interview, accepted or declined the offer to interview, were ranked, and if they matched to the program. We calculated geographic distance in miles using the Euclidean distance from the center of the applicant’s permanent zip code and the zip code of the residency program using the zipcitydistance function in SAS 9.4 (SAS Institute, Inc, 2012, Cary, NC). We excluded zip codes that were outside of the continental United States. Applicants were considered URM if they indicated they were Black/African American, Hispanic/Latino/of Spanish origin, or American Indian/Alaskan Native/Native Hawaiian/Pacific Islander.16-21 One program censored race and ethnicity from their applications in 2018 and 2020 and one program censored race and ethnicity for 2020 and 2021. The residency programs also provided information on the  number of positions available and number of positions filled each year. We calculated the total fill rate.

We used a Mann-Whitney U test to determine if there was a difference in geographic distance from applicants who applied during in-person years compared to virtual years. We used χ2 tests of independence to compare proportions of applicants’ characteristics (eg, URM and country of medical school) who applied during in-person years compared to virtual years. We used the same process to compare those who were invited to interview, those who declined to interview, and those who interviewed in both cycles.

We used mixed models with unstructured covariance matrices, where applicants were nested within institutions, to compare the distribution of applicants’ characteristics (ie, distance from permanent zip code to institution, US medical school, and URM) by time frame (in-person years vs virtual years) while accounting for the clustered nature of the data set. We used a linear mixed model for the distance from permanent zip code to institution. We used a generalized linear mixed model with a binary distribution and logit link when the outcome of interest was US medical school or URM to calculate the odds ratio and 95% confidence intervals. When the programs with select variables censored by year, we assumed the within-program variability was homogenous, thus we did a complete case analysis. To further evaluate differences across time frames, models were repeated in the subset of applicants that were offered to interview, declined to interview, and actually interviewed. We chose not to adjust for additional covariates because this study was descriptive in nature and we were not attempting to identify causal factors.

We conducted analyses using SAS, version 9.4 (SAS Institute Inc, 2012, Cary, NC). The North Texas Institutional Review Board (IRB), the ProHealth Care IRB, and the University of Mississippi Medical Center IRB all approved this study. The University of Toledo and St Marks IRBs approved the projects through an alliance agreement with the North Texas IRB.

For in-person interview cycles (3 years), there were 15,433 applicants, of which 1,706 were offered an interview, and 1,371 accepted the interview. For virtual interview cycles (2 years), there were 9,838 applicants, of which 1,197 were offered an interview, and 985 accepted the interview. The programs varied in size (Table 1). Some programs experienced a decrease in the proportion of declined interviews, while others did not.

The average distance between applicants and programs was not statistically different when comparing in-person years to virtual years (768 miles vs 772 miles; P=.27; Table 2). The same was true when comparing applicants who were offered interviews (485 miles vs 487 miles, P=.44) and applicants who actually interviewed with programs (446 miles vs 459 miles, P=.06). In the mixed model, we found a statistically significant difference in the mean distance between applicants and programs during in-person years vs virtual years, with applicants living an average of 21 miles closer to programs during virtual years (β₁=-21 95% CI: -36, -6.2; P=.006; Table 3).

Among applicants who were offered an interview, those who declined the offer were farther away than those who accepted the offer. This was true for both in-person years (637 vs 446; P<.0001) and virtual interview years (605 vs 459; P=.0005; Table 2).

Approximately 3,754 applicants (15%) did not have a permanent zip code in the United States, with similar proportion in the in-person years (2,257 or 15%) compared to the virtual years (1,497 or 15%). When stratified by those who were offered an interview, the virtual years had lower proportion with a zip code outside the continental US (2.0% vs 3.81%, P=.0055). Similarly, when stratified by those who actually interviewed, the virtual years had lower proportion with a zip code outside the continental US (2.2% vs 4.5%, P=. 004).

The proportion of applications from US medical schools increased in virtual years compared to in-person years. During in-person application years, US medical students accounted for 30% of the applications, while during virtual application years US medical students accounted for 36% of the applications (P<.0001; Table 4). A similar trend was observed for applicants offered interviews and applicants who actually interviewed. US medical students accounted for 87% of the applicants offered an interview in the in-person years and 92% in the virtual years (P<.0001). US medical students accounted for 83% of those actually interviewed in the in-person years, and 90% in the virtual years (P<.0001). This was supported in our mixed-models analysis, where virtual years had higher odds of applicants coming from a US medical school compared to in-person years (Table 3).

During in-person application years, an average of 21% of applications were submitted by URM applicants, while during virtual years, the proportion was 22% (P=.01; Table 5). In our mixed model, there were 1.08 times the odds of applicants being URM in the virtual years compared to in-person years (OR=1.08; 95% CI: 1.01, 1.15; P=.03; Table 3). There were no differences between in-person and virtual application years in rates of URM applicants offered interviews, applicants who declined interviews, or applicants who were interviewed. This held true in both the χ2 test and in the mixed model.

The fill rate during the in-person years was 97%, compared to 86% during the virtual years. One program had 1 year where they did not fill in the in-person years, compared to three programs not filling during 1 of the virtual years.

Our study examines the impact of virtual residency interviews on family medicine residency applications with regard to applicant and interview process characteristics. Most interestingly, we found no meaningful geographic difference between in-person and virtual interviewing years. In fact, we found that applicants interviewed slightly (21 miles) closer to home in virtual years, although it is questionable whether this is a meaningful difference. This was contrary to initial hypotheses that applicants may apply to more geographically diverse sets of programs. Many theorized that the reduced travel and time costs associated with virtual interviewing might result in applicants applying to more distant sites. However, our data indicate that applicants followed similar geographic patterns both before and after the introduction of virtual interviewing. Interestingly, candidates were significantly more likely to actually interview at programs geographically closer to their home regardless of the year. In fact, geographical restriction was observed across in-person and virtual years with regard to acceptance of interviews, as applicants were significantly more likely to accept interviews closer to their permanent zip code, even when programs offered interviews to applicants from a wide geographic range. There are several potential explanations for this. Applicants may have limited themselves to a radius closer to their permanent address due the familiarity afforded by this proximity. Additionally, during virtual years applicants may have chosen to visit potential residency areas themselves outside of the formal interview process, and thus limited their interview radius. Regardless of the cause, our results suggest that interviewing modality plays less of a role in application and interview decisions than a commitment to a specific location or program. If residencies hope to increase the geographic diversity of their programs, they may not find success solely by offering virtual interviews. It is also worth considering that there may be benefits to recruiting applicants from communities and medical schools that are geographically closer to residency programs, particularly in family medicine, a discipline that emphasizes community engagement and community membership. For example, residents may be more likely to stay and care for the communities they train in if those communities are their own or close to their homes. Another suggested benefit of virtual interviewing is increased potential for interview equity and inclusion of an increased number of applicants from backgrounds historically URM.22 Our study found no meaningful increase in URM applicants, interviews offered, or applicants interviewed between in-person and virtual years. We did find a small but statistically significant increase in total URM applicants (1.8%) commensurate with the overall increase of URM medical school graduates (1.3%) during those years.23 Accordingly, we are not able to attribute this increase to the use of virtual interviewing, especially because it did not carry forward to an increase in URM applicants offered interviews or actually interviewed. As with geography, residencies hoping to increase their URM residency representation are not likely to achieve this aim solely through the use of virtual interviewing.

In addition to individual variables, we observed two other noteworthy findings with regard to school and program level characteristics. First, programs received a significantly higher proportion of applications from US medical schools compared to international medical schools (OR 1.46) during virtual interview years. This seems to reflect a trend of increasing number of applications per family medicine applicant from both MD and DO schools in the United States since 2017.24 It is possible that this general trend was further facilitated by unease about the interviewing process during COVID-19 and by the decreased travel and time burdens of the virtual format mentioned above.

Second, we noted that several participating sites experienced lower fill rates not consistent with historical trends in their programs. This finding is consistent with national trends in family medicine match data for 2022, which showed an overall fill rate of 90.6%, a decrease from 92.8% the year prior and the lowest fill rate since 2007.25 This lies in contrast to the increase in the number of US medical student applications. It appears that this decreased fill rate is driven by a relative increase in the total number of residency positions (35%) compared to filled positions (27%) between 2018 and 2022.25 Moreover, it appears the number of unmatched MD applicants who ranked family medicine as their first choice remained similar between virtual and in-person years. It is possible that newer residency programs disproportionately experience lower fill rates and that more established programs may not have had a significant difference in fill rates between virtual or in-person years. Alternatively, it is possible that all programs saw a higher volume of applications, but that they came from well-qualified applicants applying to more programs than necessary, given the ease of virtual application or anxiety surrounding the virtual application process. As a result, programs competing for the same pool of applicants may not have ranked enough applicants to fill. Another possibility is that programs that had less robust virtual interviewing processes may have experienced a greater decrease in fill rates between virtual and in-person years, and this accounted for the difference.

Our study should be interpreted within the context of several limitations. Our sample included five residency programs, so there may be limitations to the generalizability to all US family medicine residencies. However, our choice of five programs represent varied geographical locations, city size, program size, and program emphases (inpatient, outpatient, obstetrics, etc) and this may improve generalizability. Second, although these programs were geographically dispersed, there were likely applicants who applied to more than one of the programs studied, resulting in some duplication of applicant data. However, the relatively large number of participants (25,271) minimizes this effect. Third, although our statistical model did adjust for program, it did not adjust for other covariates of interest. Given the data harmonization across multiple institutions, with different reporting and collecting practices, our analysis was restricted to being descriptive in nature. Adjusting the models for additional covariates may have biased the results given the missing data among the covariates and unmeasured covariates. Additionally, our study was impacted by missing data. Two programs decided to censor race and ethnicity for some years in the data set, so we were missing 14% of the data for race and ethnicity. However, we assume the missing data are distributed similarly to other years for the program, which would have minimal effects on the models. In addition, our distance analysis was restricted to the continental United States to help control for outliers, so approximately 15% of the applicants were missing a permanent zip code. However, there was a similar proportion of applicants in the in-person years and the virtual years with a zip code outside the continental United States. Among the applicants actually interviewed, less than 4% of the data was missing; however, there was a slightly higher proportion missing in the in-person years compared to the virtual years (2.2% vs 4.5%).

Future studies could collect data from programs that did not censor their applicant or race/ethnicity data over the observation period to gather more accurate data related to virtual interviewing and its effects on improving diversity. Furthermore, because the programs in our study ranked the overwhelming majority of candidates they interviewed, we did not perform separate analyses on applicant rank lists. However, it is possible that meaningful differences exist between applicants interviewed and those put on rank lists during in-person and virtual years. Future research could explore this possibility. To improve generalizability, future research should include data from additional residency programs that include a wider range of characteristics.

Overall, our study indicates that the shift from in-person to virtual interviewing was not associated with a meaningful change in geographic patterns among applicants. Additionally, despite a small change in overall applicants that reflects national trends, interviewee characteristics with regard to URM status were not impacted. As such, programs should consider any potential negative or positive impact of virtual interviewing to be minor when making decisions about offering virtual or in-person interviews. Programs may wish to consider the significant increase in US medical school applications when making interview season plans and extending interview offers.