Evaluation of the Virtual Interviews for Resident Recruitment Due to COVID-19 Travel Restrictions: A Nationwide Survey of US Senior Medical Students

Residency recruitment during the 2020-2021 academic year has been greatly disrupted by the ongoing COVID-19 pandemic. In previous match cycles, qualified applicants were offered in-person interviews at prospective residency programs. This process is structured to allow applicants an opportunity to visit program sites, interact with program faculty, and socialize with the current residents. Likewise, program directors and faculty can use this time to assess the applicants’ professionalism, interpersonal skills, and overall fit for the program. However, the resultant travel restrictions and social distancing policies attributed to COVID-19 precluded applicant travel for in-person interviews during the 2020-2021 match cycle. This led to the exclusive utilization of virtual interviews for residency recruitment. This process continued for a second year during the 2021-2022 match cycle due to the ongoing pandemic .

Thus, the aim of this study was to conduct a nationwide survey of US senior medical students who participated in the 2020-2021 virtual interview match process and to assess their confidence in using a virtual platform to holistically evaluate programs and prepare their rank-order list. Results from our study will be useful for program directors, applicants, and medical school student affairs deans, as we continue to move forward and evolve our future interview processes.

This prospective study was conducted at New York Medical College School of Medicine (Valhalla, NY). The study protocol (protocol number: 14339) was reviewed by the Institutional Review Board of New York Medical College and received an IRB exemption. The objectives of this study were (1) to evaluate if residency applicants could adequately assess prospective residency programs virtually, (2) to evaluate their satisfaction with the virtual interview process, (3) to identify any potential inequalities or challenges inherent in the virtual interview process, (4) to evaluate their confidence in developing a rank-order list during the 2020-2021 match cycle, and (5) to solicit their recommendations for future virtual interview best practices.

This study was sent by email questionnaire administered through the survey tool, Survey Monkey from March 7, 2021 to March 27, 2021 to United States allopathic, fourth-year medical students participating in the 2020-2021 residency match. The survey link was sent to 153 student affairs deans of US allopathic medical schools to distribute to their fourth-year students applying for residency. The combined number of MS-4 students at the 153 institutions was 18,704. This would lend to an estimated response rate of 1.9% (357/18,704), assuming all surveys were received.

Surveys solicited demographic information, 26 4-point Likert scale questions evaluating several factors regarding the interview and matching process, two multiple choice questions, and three free-response questions. Participants had an option of remaining anonymous and were provided with explicit information stating that their responses were being utilized for research purposes. Respondents and their representative program names were deidentified upon completion and data were stored in a secure database.

Surveys were distributed to applicants after the announcement of Match results by the National Residency Match Program (NRMP). Each recipient invited to participate in this research study was incentivized by a chance to win a raffle for a $50.00 virtual gift card on completion of the survey if they provided an email contact. Participants had an option of remaining anonymous and were provided with explicit information stating that their responses were being utilized for research purposes. Respondents were deidentified upon completion and data were stored in a secure database.

The investigators (J.M.P., L.L., N.A.L.) were involved in the construction of the survey tool. J.M.P. has been an obstetrics and gynecology residency program director for 11 years and has extensive experience in resident recruitment and the interview process. She is now the senior associate dean for student affairs at a medical school; L.L. is a medical student and a future applicant, and N.A.L. is a clinical research fellowship program director with 6 years of experience in both residency and fellowship recruitment. The investigators developed survey tools based on existing literature, knowledge of current residency practices, discussions with experts in the field, and their own experience in resident recruitment. In addition, the authors utilized data from the “Results of the 2016 National Residency Match Program (NRMP) Program Director Survey,” which contains information on which factors program directors consider important in choosing applicants for interviewing as well as for ranking applicants.1 All three investigators reviewed the survey tools to provide feedback on the relevance and clarity of questions, and projected time needed to complete the survey.

The survey tool underwent a content validation process with a core group of eight medical students who were future residency applicants. Based on these results, that showed an appropriate range of variance, we further refined the survey including incorporating questions on gauging applicants’ views on interview limits and potential disparities created by the virtual interview process. We then piloted the survey tool with a group of 12 fourth-year medical students from our medical school. The results of the pilot resulted in no substantive changes in survey content, but several questions were edited for clarity based on their feedback.

We analyzed data using the Statistical Package for the Social Sciences (SPSS, IBM version 27, Armonk, NY). We analyzed Likert data by computing meansand response distributions for each question to compare self-reported participant agreement or disagreement with survey statements. For qualitative analysis of free response questions, J.M.P., L.L., and N.A.L. reviewed all qualitative data individually to identify salient themes. They pooled their initial analyses and reviewed the qualitative data together until no new themes emerged. The final illustrative quotes were chosen to represent all salient themes by majority consensus of investigators.

A total of 357 surveys were completed. More than half (189, 52.9%) of the respondents were female and 168 (47.1%) male, and 77 (21.5%) self-identified as an underrepresented minority in medicine (URM). Sixty-two of the respondents (17.3%) are members of Alpha Omega Alpha honor medical society, and 43 (11.98%) are members of the Gold Humanism Honor Society. The top-six specialties matched into by our respondents were internal medicine (62, 17.3%), pediatrics (44, 12.6%), family medicine (41, 11.2%), emergency medicine (37, 10.3%), psychiatry (29, 8.08%), and obstetrics and gynecology (28, 7.8%). Most (301, 84.3%) applied to only one specialty, while 56 (15.7%) applied to more than one.

The mean (range) number of interview offers that each respondent received was 17.27 (range 2-61), out of which 15.33 interview offers were accepted (range 2-38) and 14.57 interviews were actually completed (range 2-35). Forty-eight (13.45%) of respondents interviewed at more than one program on a single day. When asked if respondents applied to more programs because of the virtual nature of the interview cycle compared to if the cycle was in person, answers were divided as 177 (49.58%) agreed or strongly agreed and 180 (50.42%) disagreed or strongly disagreed. Most, 265 (74.23%), believed that the virtual platform allowed them to accept more interviews than they could have if they had to travel in person. About half of the respondents (53.33%) stated that they interviewed at more programs because of the virtual nature of the current interview cycle and 139 (38.94%) placed more programs on their rank-order list. Two-thirds of respondents (239, 66.95%), believed that there should be a limit on the number of interview offers an applicant can accept. The suggested average number that should be allowed as the maximum number of interviews per specialty was 25.75 (range 10-150).

Concerning the effectiveness of virtual interviews, 173 (48.46%) agreed/strongly agreed that they were confident in their ability to assess the residency programs’ clinical training strengths and weaknesses. Only 40 (11.2%) were confidently able to assess the residency programs’ facilities, and 260 (71.82%) could confidently assess the programs’ research opportunities. Less than half of the respondents (164, 45.94%) strongly agreed or agreed they could assess their fit into the program. A total of 208 respondents (58.26%) found that virtual meet and greets helped them better assess their fit for the program, while the remaining 149 (41.74%) disagreed. Interestingly, 256 respondents (71.71%) believed that they could confidently represent themselves to the program using a virtual platform, however, a similar number (253, 70.9%) felt that would have been better able to represent themselves to programs had they been able to participate in traditional face to face interviews. Most respondents, (354, 99.3%), felt they saved money using the virtual platform, with over half (215, 60.2%) believing the financial benefit was worth any trade-off of not having an in-person interview.

The majority of respondent (248, 69.47%), claimed they could confidently prepare their rank-order list at the conclusion of the interview. However, because they could not visit programs in person, some respondents ranked a program higher if it was closer to home (148, 41.43%) or affiliated with their medical school (90, 25.07%). Additionally, due to the inability to travel, most respondents relied more heavily on other factors, such as word-of-mouth referrals from friends or colleagues (280, 78.0%) and online resources (eg, Texas Star, Association of American Medical Colleges [AAMC] Careers in Medicine, NRMP, Doximity; 280, 78.0%) when preparing their rank-order list,

Regarding potential disparities introduced by virtual interviews, 145 (40.62%) of the applicants strongly agreed/agreed that virtual interviews introduce greater inequalities into the match process. Specifically, 160 (44.94%) of applicants agreed/strongly agreed with the statement that virtual interviews disadvantaged all but the most highly-qualified applicants. However, most applicants (337, 94.40%) believed that virtual interviews reduce potential financial disparities among applicants.

When asked to choose an answer to the question, “After COVID-19 travel restrictions are lifted, going forward, which format of interviewing would you prefer be implemented?” responses were “Hybrid” (240, 66.85%), “Completely in person” (66, 18.38%), “Completely virtual” (44, 12.26%), and “Other/unsure” (9, 2.51%,). Likert responses showed more variation but also favored a hybrid format (Table 1). A multiple-choice question regarding match satisfaction found that a majority of respondents (292, 81.8%) were very satisfied or satisfied with their match results, 18 (5.0%) were neither satisfied nor dissatisfied, and 47 (13.2%) were very dissatisfied or dissatisfied.

Tables 2 and 3 reveal selected applicants’ responses to two free-text questions along with key themes, indicating their level of satisfaction with the virtual interview process and potential challenges. Regarding satisfaction, when asked “What did you like best about the virtual interview process?” the key themes identified were financial savings, convenience, and equity (Table 2). Regarding challenges, when asked “What was most challenging about the virtual interview process?” key themes identified were presenting myself virtually, assessing facilities and program location, technology, potential inequities, and assessing fit (Table 3).

The applicants had several free-text recommendations and feedback directed at program directors for future virtual interview cycles (Table 4). Key themes centered on preinterview preparation, showcasing the facility virtually, structuring of the interview day and social hour, choice of platform, and inclusion of a blinded second-look option. Regarding preinterview preparation and structure of the interview day, many applicants preferred shorter interview days with modest break time between each interview. Many recommended sending out preinterview materials including recorded introductory videos and a list of frequently asked questions that could be reviewed on the applicants’ own time, rather than “having an hour-long lecture during which our cameras are on.” Applicants commented that they would like to see more videos that showcase the facility, town, and people associated with the program that feature faculty, residents, nurses, and ancillary staff.

There were many suggestions for improvement of the virtual social hours. Several applicants commented that social hours that relied heavily on a question and answer format were ineffective. Most applicants wanted the social hour to be more fun and include activities like game night and unique ice-breakers in order to foster more organic interactions. One applicant wrote,

Please focus less on our questions during the social hours and use the time to get to know us, and for us to get to know the residents. Some of these chunks of time were completely wasted on fielding questions that would be answered at a later time, and left us feeling completely unknown to the program.

Many applicants also commented that they prefer social hours that are resident-only, as they felt that inclusion of faculty members or the program director was awkward or a red-flag. A few applicants suggested more formalized training for residents on how to effectively run a virtual social hour. Interestingly, one theme that was repeatedly expressed by a majority of respondents was the inclusion of a blinded second-look option going forward. Comments revealed that many desired an option that would allow applicants the opportunity to visit selected programs after the virtual interview, but not affect their overall ranking by the program. Suggestions to accomplish this included blinding the program director or not sharing attendance with the admission committee.

Although the 2020-2021 Match cycle was unprecedented due to its exclusive use of virtual interviews, virtual interview platforms have been employed with success as early as 2000 for residency and fellowship interviews.2-8 Although many feared that the virtual match might lead to many unmatched applicants and programs, this was not the case. Statistics from the National Resident Match Program (NRMP) revealed that 19,866 US MD senior students participated in the 2020-2021 Match cycle. This was an increase of 540 more than the previous year.9 The PGY-1 match rate for US MD seniors was 92.8%, which was within historical PGY-1 match rates for US MD seniors that ranged from 92%-95%.9 Similarly for programs, only 1,927 out of 38,106 available positions were unfilled on Match Day and only 220 programs remained unfilled at the conclusion of the Supplemental Offer and Acceptance Program (SOAP).9 Although these match outcomes were consistent with previous years, several issues need to be carefully evaluated if virtual interviews are to become part of the new normal for residency recruitment.

Regarding convenience, most respondents appreciated the time saved from traveling, as well as the comfort of being able to interview from a familiar setting. In addition, the virtual format afforded more flexibility to respondents. This included the ability to schedule multiple interviews in 1 day, being able to do West Coast and East Coast interviews back to back, and less interruptions in clinical rotations (Table 2). Our Likert survey responses further revealed that almost 60% of respondents believed the convenience of using a virtual interview format was worth any trade-off of not having an in-person interview.

Several studies have documented the financial burden associated with in-person residency interviews faced by medical students.10,11 Additionally, the AAMC reported that in recent years, US Senior MD students spent on average $4,000 on in-person interview costs. The range of student expenses spanned $1,000-$13,225 depending on specialty and number of interviews.12 One advantage to all applicants is the elimination of financial barriers related to travel. This may have afforded some applicants more interview opportunities, especially those who live in geographically remote areas or are financially challenged. One applicant commented,

I don’t know how I would’ve done this interview season if it were in-person, especially for students like me that are disadvantaged financially and don’t have family’s financial support. This virtual process allowed for equity amongst all students.

Our survey respondents indicated that they found it difficult to present themselves virtually, assess program facilities and location, and determine their fit for prospective programs. Several of these challenges were associated with inherent limitations of a virtual platform. As one applicant noted, “having natural conversations over Zoom was near impossible with delays, technological issues, and not being able to meet fellow applicants or bond over this process.” Additionally, free-text responses suggested that applicants were not able to get a feel of the program’s culture, camaraderie, or vibe, with several others also stating that, “Almost all programs seemed similar or the same through a virtual platform.” Another key challenge was inability to assess fit, as less than half of our respondents surveyed felt that they could confidently assess fit virtually. Challenges included not being able to observe how residents interacted with each other, as well as with faculty and staff. Additionally, the informal interactions that might occur between interviews or during in-person social events were nonexistent. This was reflected in many of the free-text comments as shown in Table 3. Our findings were similar to a 2021 NRMP survey where applicants reported “moderate to substantial challenges” with virtual interviews specific to learning about program curriculum, engaging comfortably in group settings, and assessing programs’ commitment to diverse faculty and leadership.13

It is difficult to determine if the challenges identified are specifically inherent to the virtual interview process itself. It is possible that the respondents would have had similar challenges in an in-person format, given their limited prior experience with residency interviews. We would anticipate that some first-year residents who have visited residency programs only virtually will be dissatisfied with their residency program once they arrive and begin residency. Whether or not resident transfers will increase following a virtual interview season due to fit is a topic for future longitudinal study. This study, however, will be challenging to conduct as the factors attributed for these transfers are often multifactorial and can include change of specialty, need or desire to move to a different location, personal conflict at current institution, and performance-related nonrenewal of contract. The rates of attrition vary widely amongst residency specialties, with highest rates reported in surgical specialties with rates as high as 20%-26%.14

This year, the NRMP data showed that US MD seniors applicants’ length of rank-order lists (ROLs) increased, with a mean number of 13.98 programs, compared to the previous 6 years that ranged from 11.97-12.82.15 Several factors could have accounted for this, including the novelty and uncertainty associated with the virtual match process. An NRMP survey of 4,287 applicants regarding the 2021 virtual match showed that the top-two factors US MD senior applicants valued most when creating their ROL were “overall goodness of fit” and “interview day experience.”16 We found that almost 70% of applicants claimed they could confidently prepare their ROL at the conclusion of the virtual interviews. However, several free-text comments indicated that inability to assess fit may have influenced ranking behavior, as one applicant noted,

I ended up ranking programs higher based on objective measures alone (fellowship match [rate], elective time, call schedule) even if I did not feel like I really fit in with the residents or culture of that program.

Another applicant commented,

My rank list ended up being heavily influenced by geography and objective logistical factors, not the ‘fit,’ because I couldn’t assess it.

Our data goes further to reveal that almost two-thirds of the applicants relied more heavily on tangible factors, including word-of-mouth referrals from friends or colleagues, as well as online resources such as Texas Star, Residency Explorer, NRMP data, and Doximity when preparing an ROL.

The elimination of financial barriers may also lead to unintended consequences in the match process. The free-text responses suggested two potential issues. First, highly-qualified applicants may accept more interview offers, thereby disadvantaging the remaining applicant pool. This could potentially lead to inequities in the match process for all but the most qualified applicants. To address this, some applicants suggested setting limits or capping the number of virtual interviews one could attend. Caps have been the subject of some controversy.17 Further investigation into the consequences of capping the number of interviews is recommended. Second, we noted concern for a regional bias in which applicants found it challenging to convey genuine interest to programs that were not close to their home, family, or medical school. Encouraging increased communication between applicants and programs may be beneficial in this regard.

One limitation of this study is the small number of respondents compared to the national applicant pool. We sent surveys to a total of 153 student affairs deans at US MD medical schools. The combined number of MS-4 students at the 153 institutions was 18,704. Since we do not know how many of the student affairs deans actually forwarded the survey to their student bodies, it is challenging to determine our survey response rate, which compared to the total number of possible respondents would be 1.9% (357/18,704). Although this response rate was very low with respect to the number of students who participated in the 2021 Match nationally, our survey responses were geographically diverse and representative. Based upon the respondents’ voluntary entry of email address to enroll in the gift card raffle, we were able to determine that we received responses from at least 22 medical schools with geographic diversity including schools in the Northeast, (New York, Rhode Island, Pennsylvania, Massachusetts, District of Columbia, Vermont), West (California, Oregon), Northern United States, (Oregon), Southern United States (Texas, Louisiana, West Virginia, Florida), Midwest (Michigan, Ohio), and Pacific Islands (Hawaii). Additionally, our respondent demographics and percentage of students in the top specialties matched were similar to the national pool.18 A total of 40,830 applicants participated in the residency Match, of whom 24,468 (48.1%) were female and 8,895 (21.8%) were URM.18 Similarly, our respondents were 52.9% female and 21.5% URM. Of the 23,027 US MD seniors, 3,477 (15.1%) were American Osteopathic Association inductees, similar to our respondent population of 17.3%.18 Of the 18,435 US MD senior applicants who matched to a PGY-1 position, the top six-matching specialties were internal medicine 3,523 (19.4%), emergency medicine 1,765 (9.4%), pediatrics 1,749 (9.5%), family medicine 1,606 (8.5%), psychiatry 1,205 (6.3%), and obstetrics and gynecology 18,438 (5.9%).19 Although we have similar demographics compared to the national population, several unaccounted factors may have led to response bias including level of match outcome satisfaction, number of interviews received, as well as a potential respondent’s personal experience with the virtual match process, be it positive or negative.

Most respondents found that virtual interviews were financially beneficial. However, difficulties in assessing fit and interpersonal relationships among prospective program faculty and residents are challenges that should be addressed and improved upon. Best practice recommendations include shorter interviews, more engaging resident-led social hours without faculty present, and preinterview packages to include video representations of the program, facility, and neighborhood. The inclusion of an option for a blinded, in-person second-look visit will be challenging, although highly desired by many applicants. Consideration of these applicants’ suggestions will certainly contribute to the evolution of the postpandemic new-normal interview format.