Background and Objectives: Virtual residency interviews were widely utilized during the COVID-19 pandemic. Little is known about the effectiveness, advantages, barriers, and acceptability of virtual interviews, casting uncertainty about how interviews should be conducted after the pandemic. We conducted a survey of interviewers to inform future decisions.
Methods: We developed and implemented an online postinterview survey of interviewers representing seven residency programs and two clinical psychology programs at one midsized academic medical center. We analyzed results using descriptive statistics.
Results: Of 312 interviewers, 136 completed the survey (44% response rate). A majority rated virtual interviews as very or extremely effective in creating a comfortable setting (79%), answering interviewee questions (86%), establishing a sense of connection (59%), evaluating interviewee strengths (64%), and communicating program culture (51%). About half felt virtual interviews were not effective at all or only slightly effective for evaluating interviewee strengths via informal interactions (51%). A similar portion agreed or strongly agreed that virtual tours (44%) and social environment (50%) information were adequate. The most frequent advantages were time efficiency (81%), reduced carbon footprint (61%) and cost savings (56%). Frequent disadvantages included technological issues (21%) and caregiving duties (18%). Most interviewers (91%) thought some form of virtual interviews should be incorporated postpandemic.
Conclusions: Interviewers found virtual interviews to be effective in most aspects, and identified more advantages than barriers. The vast majority preferred incorporation of virtual interviews in the future. Virtual tours and social activities were areas for improvement.
The COVID-19 pandemic led to widespread adoption of virtual residency program interviews. Little is known about the effectiveness, advantages, barriers, and acceptability of the virtual interview format. Studies of these factors are needed to inform decisions about future interview formats.
A growing number of studies have assessed effectiveness, perceived advantages, disadvantages, and acceptability of virtual interviews.1-36 Most studies were limited to single programs or specialties. In our current study, we analyzed the results of a multiprogram, single-institution survey of interviewers’ perceptions of effectiveness, advantages, and barriers related to virtual residency interviews. We also assessed preferences for future interview seasons.
We sent an online survey to interviewers and applicants in anesthesiology, clinical psychology internship and PhD programs, emergency medicine, family medicine, general surgery, obstetrics and gynecology, pediatrics, and psychiatry programs following virtual interviews at one academic institution in the Pacific Northwest during the 2020-2021 match cycle. We previously reported survey development methods and applicant perspectives.37 The current analysis assesses interviewer responses.
As previously described, a multidisciplinary group created a survey using expert opinion and literature review for content validity. We piloted the survey among a group of 10 people to ensure clarity. The instrument assesses effectiveness, advantages, barriers, and future preferences for interview format (Appendix Box 1). Although this was not a detailed analysis of virtual interview equity, we included several equity-related items such as pregnancy, breastfeeding, dependent care, disability, and mental health conditions.
We generated descriptive statistics of survey data. All analyses were conducted using SAS 9.4 software (SAS Institute Inc, Cary, NC). This study was exempted by the Oregon Health & Sciences University Institutional Review Board.
A total of 136 out of 312 interviewers completed the survey (response rate of 44%; Table 1). Interviewers identified predominately as female (58%) and White (73%). The majority (67%) were under age 45 years, with very few respondents over the age of 55 years (13%); 83% reported they were interviewing from an urban location.
As shown in Table 2, the majority of interviewers rated virtual interviews as very or extremely effective in creating a comfortable setting (79%) and answering interviewee questions (86%). A smaller majority of interviewers rated virtual interviews as very or extremely effective for establishing a sense of connection (59%), evaluating interviewee strengths via the formal interview (64%), and communicating program culture (51%), although an additional 30% to 38% of interviewers felt virtual interviews were moderately effective in these areas. Over half of interviewers (51%) felt that virtual interviews were either not effective at all or only slightly effective for evaluating interviewee strengths via informal interactions. Similar portions of interviewers agreed or strongly agreed the virtual format was effective at providing adequate tours (44%) and providing adequate information about the institutional social environment (50%).
Advantages and Barriers
A large portion of participants identified more advantages than barriers of virtual interviews (Figure 1). Top-perceived advantages were time efficiency (81%), reduced carbon footprint (61%), cost savings (56%), and less burdensome to schedule (49%). Interviewers selected technological issues (21%), caregiving duties (18%), access to an appropriate interview setting (16%), and increased stress/anxiety (9.9%) as top barriers. Few selected advantages or barriers related to disabilities (full data available in Appendix Table 1).
Most interviewers (91%) preferred to incorporate a virtual format in future interview cycles. The most common preference selected was “Two stages: Interviews are conducted virtually with programs having the option of inviting a subset of applicants for subsequent face to face visits” (49%). Some preferred providing an option of virtual or in-person (29%) or remaining entirely virtual (13%). The least-preferred format was entirely face to face (9%).
Our study demonstrated an overall positive perception of the virtual interview platform. Most interviewers rated virtual interviews as effective in most aspects, although the virtual format was relatively less effective for evaluating interviewee strengths via informal interactions, providing adequate tours of the facilities, and providing adequate information about the institutional social environment. Interviewers endorsed advantages more frequently than barriers, with positive perceived impacts on time efficiency, carbon footprint, cost, and scheduling burden. The majority supported virtual platforms for the future, with many endorsing a two-tiered approach (eg, virtual followed by in-person for a selected few).
Our study results are in line with a growing body of literature suggesting that a majority of interviewers find the virtual platform effective and acceptable.1-36 Concerns expressed in most studies related to providing information to applicants about the physical and social environment of a program, and an ability to assess applicants’ interpersonal interactions. The majority of studies available for comparison were single program or single specialty, whereas our study included several specialties. Results were still consistently supportive of virtual platforms.
Few studies have identified specific advantages and barriers to the virtual interview format.7,8,14,16,20,21,34 Our findings identified cost savings and time efficiency as significant advantages,in line with other studies. Novel to our study, carbon footprint savings was the second-most selected advantage. This is an important factor to consider during the current climate crisis, and in line with studies showing substantial carbon emissions related to interview travel.38-41 We uniquely identified caregiving duties and access to appropriate interview setting as top barriers, warranting future, more detailed, exploration into these concerns.
Limitations include that this was a single institution survey and had a response rate of 44%. The majority of respondents were under age 45 years, and as such may have biases related to comfort with the use of technology. Our inclusion of equity-related items was limited, potentially missing important issues such as impacts of racial bias. We also did not directly compare virtual interviews to the alternatives of in-person or hybrid interviews.
Overall, interviewers perceived virtual interviews as acceptable, effective, and advantageous with a positive impact on time efficiency, cost and carbon footprint. The majority indicated that they preferred virtual interviews be part of future recruitment cycles. Further research may focus on methods to improve evaluation of applicants’ informal interactions and tours/social activities from a virtual platform, and how to ensure equitable match results for applicants.
Presentations: Partial data from this study were presented at the Society of Teachers of Family Medicine annual meeting (virtual) in May, 2021; at the Council on Resident Education in Obstetrics and Gynecology; and at the Association of Professors of Gynecology and Obstetrics Annual Meeting, March 2022.
- Aljamaan F, Alkhattabi F, Al-Eyadhy A, et al. Faculty members’ perspective on virtual interviews for medical residency matching during the COVID-19 crisis: a national survey. Healthcare (Basel). 2021;10(1):16. doi:10.3390/healthcare10010016
- Allister L, Baghdassarian A, Caglar D, et al; American Academy of Pediatrics Section on Emergency Medicine Subcommittee of Fellowship Directors. Pediatric emergency medicine fellowship directors’ 2021 collective statement on virtual interviews and second looks. Pediatr Emerg Care. 2021;37(11):585-587. doi:10.1097/PEC.0000000000002562
- Asaad M, Elmorsi R, Ferry AM, Rajesh A, Maricevich RS. The experience of virtual interviews in resident selection: a survey of program directors in surgery. J Surg Res. 2022;270:208-213. doi:10.1016/j.jss.2021.09.011
- Brueggeman DA, Via GG, Froehle AW, Krishnamurthy AB. Virtual interviews in the era of COVID-19: expectations and perceptions of orthopaedic surgery residency candidates and program directors. JBJS Open Access. 2021;6(3):e21.00034. doi:10.2106/JBJS.OA.21.00034
- D’Angelo JD, D’Angelo AD, Mathis KL, Dozois EJ, Kelley SR. Program director opinions of virtual interviews: whatever makes my partners happy. J Surg Educ. 2021;78(6):e12-e18. doi:10.1016/j.jsurg.2021.04.008
- DiGiusto M, Lupa MC, Corridore M, Sivak EL, Lockman JL. Impact of the COVID-19 pandemic on the 2020 pediatric anesthesiology fellowship application cycle: A survey of program directors. Paediatr Anaesth. 2022;32(3):471-478. doi:10.1111/pan.14336
- Elmorsi R, Asaad M, Ferry AM, Rajesh A, Maricevich RS. How real is a virtual interview? Perspectives of orthopaedic surgery residency directors. Eur Rev Med Pharmacol Sci. 2021;25(24):7829-7832. doi:10.26355/eurrev_202112_27629
- Frohna JG, Waggoner-Fountain LA, Edwards J, et al; Pediatrics Recruitment Study Team. National pediatric experience with virtual interviews: lessons learned and future recommendations. Pediatrics. 2021;148(4):e2021052904. doi:10.1542/peds.2021-052904
- Han AY, Obiri-Yeboah D, French JC, Lipman JM. The virtual recruitment onion: peeling back the layers of the interview season during the COVID-era. J Surg Educ. 2022;79(1):77-85. doi:10.1016/j.jsurg.2021.08.003
- Hill MV, Ross EA, Crawford D, et al. Program and candidate experience with virtual interviews for the 2020 Complex General Surgical Oncology interview season during the COVID pandemic. Am J Surg. 2021;222(1):99-103. doi:10.1016/j.amjsurg.2020.11.007
- Ho G, Davis J, Hindle AK, Heinz E. Virtual residency interviews: a survey of anesthesiology program director perspectives amidst the COVID-19 pandemic. J Educ Perioper Med. 2021;23(4):E674. doi:10.46374/volxxiii_issue4_Ho
- Jimenez AE, Khalafallah AM, Romano RM, et al. Perceptions of the virtual neurosurgery application cycle during the coronavirus disease 2019 (COVID-19) pandemic: a program director Survey. World Neurosurg. 2021;154:e590-e604. doi:10.1016/j.wneu.2021.07.078
- Jones HM, Ankem A, Seroogy EA, et al. Impact of COVID-19 on radiology residency selection process: a survey of radiology residency programs in the US. Acad Radiol. 2022;29(5):779-785. doi:10.1016/j.acra.2021.12.033
- Kai A, Park JH, Bay C. The effectiveness of virtual interviews for postgraduate orthodontic residencies in North America. J World Fed Orthod. 2022;11(2):74-79. doi:10.1016/j.ejwf.2021.11.005
- Kamboj AK, Raffals LE, Martin JA, Chandrasekhara V. Virtual interviews during the COVID-19 pandemic: a survey of advanced endoscopy fellowship applicants and programs. Tech Innov Gastrointest Endosc. 2021;23(2):159-168. doi:10.1016/j.tige.2021.02.001
- Kamel S, Wang MX, Guccione J, et al. Analyzing the landscape of the 2021 virtual match: a nationwide survey of radiology programs’ stakeholders. Acad Radiol. 2021. doi:10.1016/j.acra.2021.10.029
- Kasle DA, Torabi SJ, Izreig S, Rahmati RW, Manes RP. COVID-19's Impact on the 2020-2021 resident match: a survey of otolaryngology program directors. Ann Otol Rhinol Laryngol. 2021;130(7):666-673. doi:10.1177/0003489420967045
- Lewit R, Gosain A. Virtual interviews may fall short for pediatric surgery fellowships: lessons learned from COVID-19/SARS-CoV-2. J Surg Res. 2021;259:326-331. doi:10.1016/j.jss.2020.09.029
- Mohanty A, Caldwell DJ, Hadley CC, Gibson A, Ravanpay A, Patel AJ. Virtual interviews in neurosurgery resident selection-a work in progress. World Neurosurg. 2021;155:e412-e417. doi:10.1016/j.wneu.2021.08.074
- Moran SK, Nguyen JK, Grimm LJ, et al. Should radiology residency interviews remain virtual? Results of a multi-institutional survey inform the debate. Acad Radiol. 2021. doi:10.1016/j.acra.2021.10.017
- Peyser A, Gulersen M, Nimaroff M, Mullin C, Goldman RH. Virtual obstetrics and gynecology fellowship interviews during the coronavirus disease 2019 (COVID-19) pandemic: a survey study. BMC Med Educ. 2021;21(1):449. doi:10.1186/s12909-021-02893-4
- Ponterio JM, Keslar M, Lakhi NA. The virtual interview format for fellowship recruitment in obstetrics and gynecology: a nationwide survey of program directors. Med Educ Online. 2022;27(1):2054304. doi:10.1080/10872981.2022.2054304
- Rajesh A, Asaad M, Elmorsi R, Ferry AM, Maricevich RS. The virtual interview experience for MATCH 2021: a pilot survey of general surgery residency program directors. Am Surg. 2021;31348211038555:31348211038555. doi:10.1177/00031348211038555
- Ream MA, Thompson-Stone R. Virtual residency interview experience: the child neurology residency program perspective. Pediatr Neurol. 2022;126:3-8. doi:10.1016/j.pediatrneurol.2021.09.016
- Relke N, Soleas E, Lui J. Internal medicine residents’ and program directors’ perception of virtual interviews during COVID-19: a national survey. Can Med Educ J. 2021;12(2):e129-e130. doi:10.36834/cmej.71041
- Rhoades JS, Ramsey PS, Metz TD, Lewkowitz AK. Maternal-fetal medicine program director experience of exclusive virtual interviewing during the coronavirus disease 2019 pandemic. Am J Obstet Gynecol MFM. 2021;3(4):100344. doi:10.1016/j.ajogmf.2021.100344
- Robinson KA, Shin B, Gangadharan SP. A comparison between in-person and virtual fellowship interviews during the COVID-19 pandemic. J Surg Educ. 2021;78(4):1175-1181. doi:10.1016/j.jsurg.2020.11.006
- Romano R, Mukherjee D, Michael LM, et al; Society of Neurological Surgeons. Optimizing the residency application process: insights from neurological surgery during the pandemic virtual application cycle. J Neurosurg. 2022;•••:1-9. doi:10.3171/2021.11.JNS211851
- Sarac BA, Shen AH, Nassar AH, et al. Virtual interviews for the integrated plastic surgery residency match: the program director perspective. Plast Reconstr Surg Glob Open. 2021;9(7):e3707. doi:10.1097/GOX.0000000000003707
- Shreffler J, Platt M, Thé S, Huecker M. Planning virtual residency interviews as a result of COVID-19: insight from residency applicants and physicians conducting interviews. Postgrad Med J. 2022;98(1158):276-280. doi:10.1136/postgradmedj-2020-139182
- Tawfik AM, Imbergamo C, Chen V, et al. Perspectives on the orthopaedic surgery residency application process during the COVID-19 pandemic. J Am Acad Orthop Surg Glob Res Rev. 2021;5(10). doi:10.5435/JAAOSGlobal-D-21-00091
- van der Laan L, George R, Nesiama JA, et al. Virtual interviewing for pediatric emergency medicine fellowship-a national survey. Pediatr Emerg Care. 2021. doi:10.1097/PEC.0000000000002549
- Venincasa MJ, Steren B, Young BK, et al. Ophthalmology residency match in the Covid-19 Era: applicant and program director perceptions of the 2020-2021 application cycle. Semin Ophthalmol. 2022;37(1):36-41. doi:10.1080/08820538.2021.1906917
- Vining CC, Eng OS, Hogg ME, et al. Virtual surgical fellowship recruitment during COVID-19 and its implications for resident/fellow recruitment in the future. Ann Surg Oncol. 2020;27(S3)(suppl 3):911-915. doi:10.1245/s10434-020-08623-2
- Wang KY, Babu J, Zhang B, Jami M, Musharbash F, LaPorte D. Effect of the COVID-19 pandemic on the orthopaedic surgery residency application process: what can we learn? J Am Acad Orthop Surg Glob Res Rev. 2021;5(10). doi:10.5435/JAAOSGlobal-D-21-00204
- Warren CJ, Reitblat CR, Ferreri CA, et al. Virtual “matchmaking”: stakeholder perspectives on the future of the urology residency match. Urology. 2022;164:11-17. doi:10.1016/j.urology.2022.02.016
- Domingo A, Rdesinski R, Stenson A, et al. Virtual residency interviews: applicant perceptions regarding virtual interview effectiveness, advantages, and barriers. Journal of Graduate Medical Education. [Accepted, publication pending April 2022] doi:10.4300/JGME-D-21-00675.1
- Donahue LM, Morgan HK, Peterson WJ, Williams JA. The carbon footprint of residency interview travel. J Grad Med Educ. 2021;13(1):89-94. doi:10.4300/JGME-D-20-00418.1
- Bernstein D, Beshar I. The carbon footprint of residency interviews. Acad Med. 2021;96(7):932. doi:10.1097/ACM.0000000000004096
- Liang KE, Dawson JQ, Stoian MD, Clark DG, Wynes S, Donner SD. A carbon footprint study of the Canadian medical residency interview tour. Med Teach. 2021;43(11):1302-1308. doi:10.1080/0142159X.2021.1944612
- Fung BS, Raiche I, Lamb T, Gawad N, MacNeill AJ, Moloo H. A chance for reform: the environmental impact of travel for general surgery residency interviews. Can Med Educ J. 2021;12(3):8-18. doi:10.36834/cmej.71022