Virtual Recruitment Effects on Matched Residents in Family Medicine: Experiences From Central Pennsylvania

Residency candidate interviews are an important element of candidate selection, as they can provide a way to capture an applicant’s personal characteristics, desires, and motivations for residency.^1-4 Prior to 2020, interviews were typically conducted in-person, however, due to the COVID-19 pandemic, candidate interviews were conducted virtually during the 2020-2021 application cycle.^{5, 6} The Coalition for Physician Accountability, American Academy of Family Physicians, Society of Teachers of Family Medicine, Association of Departments of Family Medicine, North American Primary Care Research Group, and Association of Family Medicine Residency Directors all recommended universal virtual interviews for the 2021-2022 interview season as well.^{7, 8}

Recent research has reported that the setting in which a candidate interview takes place (virtual, in-person, or hybrid) has an impact on recruitment and match outcomes for family medicine residency programs.^9-18 While virtual recruitment has been mostly well received by applicants, particularly the financial benefit, they find the format difficult in assessing fit, morale, camaraderie, and the overall resident experience.^{9, 12, 14, 15, 17} They also reported preference for shorter interviews, more engagement with current residents, and adequate website information including video representation of program facilities.^{10, 14, 18} On the other hand, program directors reported concerns about bias,¹⁶ and did not feel that the financial benefits to students was reason enough to continue virtual interviews.¹³ There is little research comparing family medicine match outcomes before and after the transition to the virtual format, and only one is known to examine applicant geographic and demographic data.¹¹

This study evaluated the impact of virtual interviews by comparing a baseline of in-person resident interviewing outcomes from 2015-2019 to virtual interviewing outcomes using data from the 2020-2022 virtual interview seasons in three family medicine residencies. Analyzed outcomes of matched residents were comprised of data that the programs had stored including percentage of filled positions, position on program rank list, distance from residency program, and self-disclosed underrepresented in medicine (URiM) demographic information. We hypothesized that virtual candidate interviews were as effective as in-person interviews.

We completed retrospective analysis on data reported in the Match Results by Ranked Applicants report provided from 2016-2022 (representing interview seasons of 2015-2022) from the National Resident Matching Program. Data were compiled across a convenience sample of three family medicine residency programs affiliated with Penn State Health: (1) the M.S. Hershey Medical Center in Hershey, Pennsylvania (Institution 1), a 3-year, university-based program, recruiting eight residents per class, (2) the Mount Nittany Medical Center in State College, Pennsylvania (Institution 2), a 3-year, community based, university-affiliated program, recruiting six residents per class, and (3) Saint Joseph Hospital in Reading, Pennsylvania (Institution 3), a 3-year community based, university-affiliated program recruiting six residents per class. Table 1 provides detail on resident demographics for each site. All in-person interviews were conducted at all programs from 2015-2019. All virtual interviews were required at all programs from 2020-2022. This study was approved by the institutional review board at Penn State University.

Primary outcome measures were program fill (filled vs unfilled), average position on rank list, average distance from residency site to matched applicant current and permanent address zip code, and URiM self-disclosed demographic information.  

In-person interview data 2016-2019 was compared to virtual interview data from 2020-2022 using a χ² or Fisher Exact test for filled versus unfilled and URiM percentage of matched resident. We used 2-tailed t test for average position on the rank list and average distance from current and permanent address. All tests were performed to 95% confidence.

Pre-COVID-19, in-person data from 2016-2019 included 31, 34, and 40 total residents from Institution 2, Institution 3, and Institution 1, respectively. Virtual data from 2020-2022 included 12, 14, and 17 total residents from Institution 2, Institution 3, and Institution 1, respectively. Institution 3 had no ranking data from 2016-2019.

Results are summarized in Table 2. Only Institution 3’s program had a significant difference in filled positions in-person versus virtual (97.1% vs 64.3%; P=.0058). Institution 2’s program matched applicants from a significantly closer current address (615 mi vs 250 mi; P=.048). There was no significant difference across all programs in average position of matched applicant on the program’s rank order list, distance between the program and applicants’ current address, or percentage of URiM matched residents.

These results suggest that the impact of virtual interviews on match decisions in family medicine is likely site-specific and generally small.

Only Institution 3 had a significant difference in fill rates; nationally, fill rates for family medicine programs ranged from 10-24% during the study period.^{19, 20} However, that program underwent significant structural changes during its first virtual interview season. The other programs' outcomes—no decrease in fill rates with virtual interviews—are likely more generalizable.

We found no differences for any site in distance from permanent address, although Institution 2 had a difference in distance from current address. This is consistent with reports that geographic location of a residency program is the most important factor in student decision-making when selecting a residency program¹⁸; this is not expected to change by a virtual interview experience.²¹ 

Institution 3 did not have ranking data available for the in-person interviews for comparison because of significant program leadership turnover that occurred including the program director and residency coordinator positions. Institution 1 had a lower average position on the rank list, but this was not significant. Insitution 2 had a slight increase in average position, but this too was not significant. Given the typical range of candidate ranks, it is likely that most programs will not experience significant rank changes. However, programs with very high or very low average positions may have a different experience.

As a retrospective study of three residency programs part of a large academic health system in Central Pennsylvania, we were unable to control for a myriad of factors that may influence program matches, limiting generalizability of our regional-specific results to other US residency programs. Both Institution 1 (2019) and Institution 3 (2020) changed program directors within the study period, which may have affected applicant perception of the stability of the programs, thus affecting recruitment data. The change in leadership (program director and coordinator) also resulted in a limitation to the available rank list data for Institution 3’s program. A single department chair, who also changed during the study period (2017) administers all three sites. While more remote, this too may have affected perception of the turnover of program leadership, and thus affected recruitment. Although no significant difference was demonstrated, the number and percentage of URiM matched residents was also low; further investigation of virtual interviewing effects on graduate medical education diversity, equity and inclusion is warranted.

Strengths of the study include its use of three different sites, objective outcome data, and inclusion of 5 years of in-person data and 2 years of virtual data.

Our data suggest that the impact of virtual interviewing on family medicine match data is minimal. These conclusions apply to the programs that were studied; further research is needed to confirm the generalizability of these results.