What’s Worth Doing in Virtual Recruitment? A Regional Survey of Program Directors and Incoming Interns

The pressure is high for family medicine residency programs to match with residents that are compatible with their program, institution, patient population, and community. Therefore, programs invest a great deal in recruiting and interviewing medical students for their training programs. Prior to 2020, programs met students by attending conferences and hosting recruiting events. They held in-person interviews where students engaged with the program’s faculty, residents, and staff; toured facilities and the community; and were invited to informal dinners or social events. However, in 2020, with rising cases of COVID-19 and the recommendations from the Coalition for Physician Accountability,1 including the Association of American Medical Colleges (AAMC), residency programs quickly pivoted to virtual recruitment and interviews for the incoming class of 2021.

Given this rapid transition and the relative novelty of adapting virtual modalities for resident recruitment purposes, little is known about the processes or outcomes of virtual recruitment. Some programs were conducting virtual interviews prior to the COVID-19 pandemic,2 and the AAMC and others worked quickly to compile resources and tips for programs.3 Findings in the recent literature examined the use of social media4 and smartphone apps5 to aid in recruitment, offered best practices for communication during web-based interviews,6 and identified recommendations for avoiding bias,7 though even these resources were not available at the onset of the virtual recruitment recommendation.1 However, as of the time we conducted this study, there are no published data on the breadth of implemented recruitment and interviewing activities in family medicine residency programs, nor data on how students perceived the utility of these activities.

Our objectives were three-fold: (1) describe the variety and value of the virtual recruitment activities our programs engaged in during the virtual recruitment season, (2) identify which activities were most and least helpful from the student perspective, and (3) assess how well our programs anticipated the interests and needs of their student applicants in the virtual recruitment activities they offered.

The Family Medicine Residency Network (FMRN) comprises 31 family medicine residency programs and 10 rural training tracks across the five-state region of Washington, Wyoming, Alaska, Montana, and Idaho (WWAMI). Across all training years there are more than 700 family medicine residents who are part of FMRN programs. To learn about the experiences of our programs/program directors and of the incoming interns who participated in the 2020-2021 virtual interview season, we conducted simultaneous surveys of these two populations after the 2021 Match.

We began our survey development process in March 2021 by asking our regional program directors about the activities they did in the 2020-2021 recruitment season because of the virtual interview process. We used this list to formulate the response options for our primary questions of interest on both surveys.

In their survey, program directors were asked about which virtual recruitment activities their program offered, which of these activities that involved interacting with students were most and least helpful in creating their rank list, and which they thought most positively impacted medical students’ perception of their program. In order to gauge the program directors’ perception of the impact of the virtual recruitment season on the Match, we also asked how they fared in the Match compared to previous years.

Using the same list of activities, the survey for the incoming interns asked which activities they participated in for any program during recruitment season, which activities were most and least helpful in deciding their rank list, and which impacted their perception of a program. We also asked an open-ended question about other factors that impacted their decision on how to rank programs. We pilot tested this survey with a current resident and three fourth-year medical students for readability and flow before sending it.

We sent both surveys electronically on June 1, 2021 and we sent weekly reminders via email until July 15, 2021. The program survey was sent to 49 program director and rural training track (RTT) site director email addresses; six were determined to be duplicates and were removed from the sample for a final program director sample of 43. We sent the incoming intern survey to 242 email addresses. We collected, stored, and managed all data via REDCap (Research Electronic Data Capture)8 hosted at the University of Washington. We analyzed our results with descriptive statistics. The University of Washington Human Subjects Division deemed this study exempt from Institutional Review Board review under category 2 of the IRB exemption criteria.

Sixty-nine percent (167/242) of incoming interns responded to the survey. Of the 35 activities identified by FMRN program directors, more than half of the incoming intern respondents participated in 15 (Table 1). The activities respondents found most and least helpful to deciding their rank list are also included. Of the 15 most common activities, virtual interviews and virtual get-togethers with residents were chosen most often as helpful to deciding their rank list by those who had participated in that activity. In contrast, the activities that were least helpful in deciding the respondent’s rank list were receiving gifts and meals.

The activities that fewer than half of respondents participated in are listed in Table 2. Among these activities, participating in an in-person interview as a rotating sub-I was rated as most helpful to deciding rank list by more than half of those who did this activity. Participating in a didactics open house and looking for a program on social media (Facebook, TikTok, and Twitter, specifically) were rated as least helpful to deciding rank list by more than half of those who did these activities.

When asked about other factors that impacted how they decided to rank programs (free text), incoming intern respondents most frequently cited the importance of a program’s location (n=44). Other repeated responses included curriculum or elective offerings and opportunities (n=16), gut feeling/felt like a good fit (n=9), communication/responsiveness of program (n=7), quality of life or work/life balance (n=5), previous experience with a program such as a sub-internship (n=5), reputation (n=4), patient population (n=3), payment/benefits/cost of living (n=3), and diversity of residents (n=3).

Thirty-four of the 43 directors (79%) responded to the survey. Eleven of the 35 activities listed in the survey were offered by at least half of the FMRN programs (Table 3). After removing the 13 activities on the overall list that did not involve directly interacting with students (eg, creating a recruitment video, updating program’s website, offering a prerecorded tour, increasing social media presence, sending a gift bag, etc), program director respondents ranked which activities they did and found to be most and least helpful in creating their rank list (Table 3).

Sixty-two percent of program director respondents reported that they matched as expected, better than, or much better than they usually do on their 2021 rank list, though more reported that they matched further down their rank list (38%) than those who reported they matched at a higher spot on their list (30%, Figure 1).

In addition to sharing about what positively and negatively impacted their rank list, incoming intern respondents shared what positively impacted their perception of a program. These were similar to what these respondents found most helpful in creating their rank list (virtual interviews with residents, program directors, and faculty; looking at a program’s website; resident-only get-togethers) but their perception was also impacted by looking at resident bios, watching recruitment videos, and looking for a program on Instagram. Program directors were also asked which activities they thought most positively impacted medical students’ perception of their program. Table 4 shows a comparison of rank order between program director and incoming intern responses to these questions for the activities that more than half of each group participated in.

From the incoming intern respondents, we learned that a program’s website and resident-only social time with applicants are the most helpful in determining rank list, though timing of this get-together (before or after interviews) is less important. Having interviews with not only residents and faculty, but specifically the program director, was also ranked of high importance. Location of a program was the most frequently cited factor outside of recruitment activities that contributed to rank decisions.

The programs in the FMRN participated in a wide range of recruitment activities during the 2020-2021 virtual recruitment season. After the interviews themselves, opportunities for applicants to engage in resident get-togethers was the most important factor that impacted a program’s rank list, so any additional live networking time that can be offered with the current residents was considered valuable. Although more FMRN programs reported that they fared the same or better in the Match following the virtual interview cycle than those who reported going lower than usual on their rank list, a sizable number of programs (more than one third) reported going more than 10% lower down their rank list during the 2021 Match than a typical year. In a supplemental analysis, we saw no difference between programs that fared worse by program size, accreditation year, or rurality or program location.

While more than 25% of FMRN respondents reported spending money on gift bags or food delivery, these were ranked as the least helpful recruitment strategies by students. Given that the average FMRN program interviews 15 applicants per position, assuming an average cost of $20 for a meal or gift bag, discontinuing this offering could lead to several thousand dollars of savings for programs. Prerecorded hospital/clinic tours, which may incur costs for video equipment and editing, were also noted to be among the least helpful to applicants. Strategies with lower financial investment, such as updating the program website and increasing opportunities for resident-only engagement as discussed above, may be more cost effective and ultimately more beneficial to applicants and programs.

When comparing what program directors thought would impact applicant perception of a program to what incoming interns actually reported impacted their perception, we found that program director perception generally matched applicant reports. However, program directors overestimated the importance of the recruitment video and a prerecorded hospital/clinic tour and underestimated the importance of resident-only social interactions.

We were surprised to learn that residency fair participation had little to do with program perception or ranking by students because feedback from FMRN residents who had attended the in-person American Academy of Family Physicians National Conference has historically been very positive. However, the virtual residency fair experience may not be typical of past years and experiences. More importantly, our survey was designed to focus on factors that specifically impact the creation of the rank list. The impact of upstream factors, or factors that help students decide which programs to apply to, like a residency fair, are not captured in our study. It is possible that other factors that were also ranked low in our survey like resident bios, recruitment videos, and prerecorded hospital/clinic tour, are upstream factors and their role may be more important in helping applicants decide which programs to apply to.

Although we had large response from both our program directors and the incoming interns, due to the regional nature of the FMRN, our study results may not be broadly applicable to other parts of the country. However, the programs that make up the FMRN vary widely by size, sponsor, location, and mission, and offered very different virtual interview opportunities, so most bias due to program similarity would be attenuated. Additionally, since we used incoming interns as our interviewee sample, there is likely response bias since we do not know about the opinions or experiences of other applicants. We were most interested in what students found helpful that would align with our programs’ experiences; a study examining engagement and perception of the interview process among all students who participated in their interview process might find something different. However, most of our questions were general enough to have applied to most programs and students participating in the 2020-2021 virtual interview season.

As we move into a future where virtual interviews may continue to be the norm or at least an option for many,9 our study provides insight to residency programs on the types of recruitment strategies that students are most likely to find most valuable in their rank order discernment. In particular, we recommend setting up the virtual interview process to maximize interactions with current residents and to make sure all interviewees have a chance to interview with multiple individuals in different positions across the program.