Background and Objectives: As a result of the COVID-19 pandemic, interviews during the 2021 US residency match were conducted virtually, a practice again recommended and repeated by many programs in 2022. The impact of virtual interviews on recruitment and match outcomes has recently been of interest, with results showing the virtual format to be mostly well received by applicants due to cost, travel, and scheduling benefits. Few studies have looked at pre/posttransition comparisons of applicant geographic and demographic data. We compared objective match outcomes between in-person and virtual interviews across three residency programs.
Methods: We conducted a retrospective cross-sectional analysis of National Residency Matching Program data between 2015-2022 across three family medicine residency programs. Primary outcomes were fill rate, average rank position, distance from program, and percentage of underrepresented in medicine demographic status for matched applicants. We compared aggregate in-person data (2015-2019) to aggregate virtual data (2020-2022) for each program using χ2, Fisher Exact test, or 2-tailed t tests to 95% confidence.
Results: Saint Joseph Hospital in Reading, Pennsylvania, a 3-year community-based university affiliated program, had significantly more unfilled positions during virtual recruitment (P=.0058). Mount Nittany Medical Center in State College, Pennsylvania, a 3-year community based university-affiliated program, had a significant difference in distance of matched residents’ current address (P=.048). Virtual interviews were not associated with significant differences in average position on rank list, average distance from permanent address zip code, or percentage of underrepresented in medicine (URiM) demographic status for matched applicants.
Conclusions: The impact of virtual interviewing on unfilled positions and geographic data is likely site specific and generally small, as some programs had significant structural changes. Further research is needed to confirm the generalizability of these results and explore future comparisons of demographic and geographic characteristics of matched applicants pre/posttransition to the virtual format.
Residency candidate interviews are an important element of candidate selection, as they can provide a way to capture an applicant’s personal characteristics, desires, and motivations for residency.1-4 Prior to 2020, interviews were typically conducted in-person, however, due to the COVID-19 pandemic, candidate interviews were conducted virtually during the 2020-2021 application cycle.5, 6 The Coalition for Physician Accountability, American Academy of Family Physicians, Society of Teachers of Family Medicine, Association of Departments of Family Medicine, North American Primary Care Research Group, and Association of Family Medicine Residency Directors all recommended universal virtual interviews for the 2021-2022 interview season as well.7, 8
Recent research has reported that the setting in which a candidate interview takes place (virtual, in-person, or hybrid) has an impact on recruitment and match outcomes for family medicine residency programs.9-18 While virtual recruitment has been mostly well received by applicants, particularly the financial benefit, they find the format difficult in assessing fit, morale, camaraderie, and the overall resident experience.9, 12, 14, 15, 17 They also reported preference for shorter interviews, more engagement with current residents, and adequate website information including video representation of program facilities.10, 14, 18 On the other hand, program directors reported concerns about bias,16 and did not feel that the financial benefits to students was reason enough to continue virtual interviews.13 There is little research comparing family medicine match outcomes before and after the transition to the virtual format, and only one is known to examine applicant geographic and demographic data.11
This study evaluated the impact of virtual interviews by comparing a baseline of in-person resident interviewing outcomes from 2015-2019 to virtual interviewing outcomes using data from the 2020-2022 virtual interview seasons in three family medicine residencies. Analyzed outcomes of matched residents were comprised of data that the programs had stored including percentage of filled positions, position on program rank list, distance from residency program, and self-disclosed underrepresented in medicine (URiM) demographic information. We hypothesized that virtual candidate interviews were as effective as in-person interviews.
We completed retrospective analysis on data reported in the Match Results by Ranked Applicants report provided from 2016-2022 (representing interview seasons of 2015-2022) from the National Resident Matching Program. Data were compiled across a convenience sample of three family medicine residency programs affiliated with Penn State Health: (1) the M.S. Hershey Medical Center in Hershey, Pennsylvania (Institution 1), a 3-year, university-based program, recruiting eight residents per class, (2) the Mount Nittany Medical Center in State College, Pennsylvania (Institution 2), a 3-year, community based, university-affiliated program, recruiting six residents per class, and (3) Saint Joseph Hospital in Reading, Pennsylvania (Institution 3), a 3-year community based, university-affiliated program recruiting six residents per class. Table 1 provides detail on resident demographics for each site. All in-person interviews were conducted at all programs from 2015-2019. All virtual interviews were required at all programs from 2020-2022. This study was approved by the institutional review board at Penn State University.
Primary outcome measures were program fill (filled vs unfilled), average position on rank list, average distance from residency site to matched applicant current and permanent address zip code, and URiM self-disclosed demographic information.
In-person interview data 2016-2019 was compared to virtual interview data from 2020-2022 using a χ2 or Fisher Exact test for filled versus unfilled and URiM percentage of matched resident. We used 2-tailed t test for average position on the rank list and average distance from current and permanent address. All tests were performed to 95% confidence.
Pre-COVID-19, in-person data from 2016-2019 included 31, 34, and 40 total residents from Institution 2, Institution 3, and Institution 1, respectively. Virtual data from 2020-2022 included 12, 14, and 17 total residents from Institution 2, Institution 3, and Institution 1, respectively. Institution 3 had no ranking data from 2016-2019.
Results are summarized in Table 2. Only Institution 3’s program had a significant difference in filled positions in-person versus virtual (97.1% vs 64.3%; P=.0058). Institution 2’s program matched applicants from a significantly closer current address (615 mi vs 250 mi; P=.048). There was no significant difference across all programs in average position of matched applicant on the program’s rank order list, distance between the program and applicants’ current address, or percentage of URiM matched residents.
These results suggest that the impact of virtual interviews on match decisions in family medicine is likely site-specific and generally small.
Only Institution 3 had a significant difference in fill rates; nationally, fill rates for family medicine programs ranged from 10-24% during the study period.19, 20 However, that program underwent significant structural changes during its first virtual interview season. The other programs' outcomes—no decrease in fill rates with virtual interviews—are likely more generalizable.
We found no differences for any site in distance from permanent address, although Institution 2 had a difference in distance from current address. This is consistent with reports that geographic location of a residency program is the most important factor in student decision-making when selecting a residency program18; this is not expected to change by a virtual interview experience.21
Institution 3 did not have ranking data available for the in-person interviews for comparison because of significant program leadership turnover that occurred including the program director and residency coordinator positions. Institution 1 had a lower average position on the rank list, but this was not significant. Insitution 2 had a slight increase in average position, but this too was not significant. Given the typical range of candidate ranks, it is likely that most programs will not experience significant rank changes. However, programs with very high or very low average positions may have a different experience.
As a retrospective study of three residency programs part of a large academic health system in Central Pennsylvania, we were unable to control for a myriad of factors that may influence program matches, limiting generalizability of our regional-specific results to other US residency programs. Both Institution 1 (2019) and Institution 3 (2020) changed program directors within the study period, which may have affected applicant perception of the stability of the programs, thus affecting recruitment data. The change in leadership (program director and coordinator) also resulted in a limitation to the available rank list data for Institution 3’s program. A single department chair, who also changed during the study period (2017) administers all three sites. While more remote, this too may have affected perception of the turnover of program leadership, and thus affected recruitment. Although no significant difference was demonstrated, the number and percentage of URiM matched residents was also low; further investigation of virtual interviewing effects on graduate medical education diversity, equity and inclusion is warranted.
Strengths of the study include its use of three different sites, objective outcome data, and inclusion of 5 years of in-person data and 2 years of virtual data.
Our data suggest that the impact of virtual interviewing on family medicine match data is minimal. These conclusions apply to the programs that were studied; further research is needed to confirm the generalizability of these results.
Presentations: This project was presented at the 50th NAPCRG Annual Meeting, November 18-22, 2022, Phoenix, Arizona.
- Data Release and Research Committee. Results of the 2016 NRMP Program Director Survey. National Resident Matching Program; 2016. Accessed March, 2023. https://www.nrmp.org/wp-content/uploads/2021/07/NRMP-2016-Program-Director-Survey.pdf
- Data Release and Research Committee. Results of the 2018 NRMP Program Director Survey. National Resident Matching Program; 2018. https://www.nrmp.org/wp-content/uploads/2021/07/NRMP-2018-Program-Director-Survey-for-WWW.pdf
- Data Release and Research Committee. Results of the 2020 NRMP Program Director Survey. National Resident Matching Program; 2020. https://www.nrmp.org/wp-content/uploads/2021/08/2020-PD-Survey.pdf
- Wagoner NE, Suriano JR, Stoner JA. Factors used by program directors to select residents. J Med Educ. 1986;61(1):10-21. doi:10.1097/00001888-198601000-00002
- Hammoud MM, Standiford T, Carmody JB. Potential implications of COVID-19 for the 2020-2021 residency application cycle. JAMA. 2020;324(1):29-30. doi:10.1001/jama.2020.8911
- Hammoud MM, Standiford TC, Carmody JB. The 2020-2021 residency application cycle: lessons learned and lingering problems. JAMA. 2021;325(22):2249-2250. doi:10.1001/jama.2021.5708
- American Academy of Family Physicians. Recommendations for 2021-2022 Family Medicine Residency Interview Process. accessed February 27 2022. https://www.aafp.org/dam/AAFP/documents/medical_education_residency/the_match/residency-interview-recs-2021.pdf
- Association of American Medical Colleges. Conducting interviews during the coronavirus pandemic. Med Educ. 2021.
- Bishop T, Heinrich L, Greenberg JB, Wenner R, Furst W, Wong J. The impact of virtual interviews on the resident candidate: a before-and-after comparison. Fam Med. 2022;54(10):833-835. doi:10.22454/FamMed.2022.510274
- Domingo A, Rdesinski RE, Cheng A, et al. Effectiveness of virtual residency interviews: interviewer perspectives. Fam Med. 2022;54(10):828-832. doi:10.22454/FamMed.2022.177754
- Grace AJ, Teigen K, Buck K, et al. Evaluating changes in family medicine applicant characteristics following the onset of virtual interviewing. Fam Med. 2022;54(10):791-797. doi:10.22454/FamMed.2022.625652
- Hays A, Khare M, Pluta D, Verzal R, Garry JP. First-year resident perceptions of virtual interviewing. Fam Med. 2022;54(10):814-819. doi:10.22454/FamMed.2022.364201
- Keister DM, Wu V, Brohm V, et al. The impact of virtual interviews on recruitment and implicit bias. Fam Med. 2022;54(10):769-775. doi:10.22454/FamMed.2022.358658
- Ponterio JM, Levy L, Lakhi NA. Evaluation of the virtual interviews for resident recruitment due to covid-19 travel restrictions: a nationwide survey of us senior medical students. Fam Med. 2022;54(10):776-783. doi:10.22454/FamMed.2022.592364
- Rohrberg T, Walling A, Gillam M, St Peter M, Nilsen K. Interviewing for family medicine residency: in-person, virtual, or hybrid? Fam Med. 2022;54(10):820-827. doi:10.22454/FamMed.2022.951860
- Snellings JE, Moore MA, Meyer D. Financial considerations after a fully-virtual interview season: a cera study of family medicine residency program directors. Fam Med. 2022;54(10):798-803. doi:10.22454/FamMed.2022.517173
- Tout W, Oyola S, Sharif Z, VanGompel EW. Applicant evaluation of residency programs in a virtual format: a mixed-methods study. Fam Med. 2022;54(10):804-813. doi:10.22454/FamMed.2022.148473
- Weidner A, Ormsby M, Shih G. What’s worth doing in virtual recruitment? a regional survey of program directors and incoming interns. Fam Med. 2022;54(10):784-790. doi:10.22454/FamMed.2022.359153
- National Resident Matching Program. Match Data & Report Archives 1984-2021. January 4, 2023. https://www.nrmp.org/match-data-analytics/archives/
- National Resident Matching Program. Residency Data & Reports. Accessed January 4, 2023. https://www.nrmp.org/match-data-analytics/residency-data-reports/
- Love JN, Howell JM, Hegarty CB, et al. Factors that influence medical student selection of an emergency medicine residency program: implications for training programs. Acad Emerg Med. 2012;19(4):455-460. doi:10.1111/j.1553-2712.2012.01323.x