Interviewing for Family Medicine Residency: In-Person, Virtual, or Hybrid?

The full impact of the COVID-19 pandemic on the National Resident Matching Program (NRMP) remains unclear, but documenting the lessons of the forced experiment in virtual interviewing during recent years is crucial to designing a more effective and efficient process.1-8 Prior to the pandemic, pressure was already mounting to reform the process in order to decrease the burden on both programs and applicants.9-25 In addition to substantial personal stress and educational disruption,26-33 applicants reported average costs of $3,000-$7,000 depending on specialty, with enormous variation among applicants to the same specialty.34-45 Some individuals reported in excess of $25,000 in residency interviewing expenses. In addition, up to 60% of students reported restricting choices about specialty and program due to application costs.46 A 2015 Association of American Medical Colleges report authored by student representatives criticized “the unnecessarily large expense in addition to the already unreasonable debt burden of medical education.”35 Financial challenges have continued to increase for medical students. A 2022 statement by trainee leaders concurred that “the current financial burden is untenable and perpetuates inequity in the system,” and cautioned against any reform of the “complex and dysfunctional system” that included additional costs for applicants.1 Controlling the costs and time required of applicants must be primary goals of any changes to the NRMP process. As virtual interviewing appears to reduce both costs and time requirements, this format seems likely to assume a larger role in reformed NRMP systems.47-50

For 2021-2022, the Coalition For Physician Accountability recommended that “all interviews should be conducted virtually… due to growing concerns about the Delta variant and potential risks and disruptions from potential future variants.”51 The virtual format was endorsed by all groups representing family medicine in a joint statement encouraging family medicine residency programs to exclusively utilize virtual interviews and avoid hybrid models as “consistency in the interview process, with most or all programs providing virtual interviews, will help create equity in the process for students.”52 The all-virtual interviewing NRMP format in 2020-2021 was associated with significant savings for applicants to all specialties from our institution. Compared to previous years, applicants reported an average 78.7% reduction in cost (from $4,529 to $963) and almost halved the time committed to interviewing (from 28.2 days to 14.9).46 Reductions in cost and time spent on interviewing have also been reported in current literature reviewing all-virtual interviewing,6,64-70 but the data is limited, especially for the specialty of family medicine. Surveyed applicants have also expressed varied feelings about the all-virtual interviewing process, with some studies noting applicant satisfaction with the process and others wishing to continue in-person interviewing.65-70 Further, some family medicine residency programs offered only in-person interviews, some only virtual interviews, and some programs offered both as a hybrid.53

This study sought to assess the impact of interview format on family medicine applicants’ cost and time spent during the NRMP residency interviewing process. Specifically, the purpose was to document the costs and time spent by family medicine residency program applicants in years using three different interview formats: traditional in-person only (2019-2020), virtual only (2020-2021), and hybrid (2021-2022). The hybrid format allowed applicants to complete either in-person or virtual interviews with residency programs based on personal preference or program availability. We hypothesized that both virtual and hybrid formats would be associated with lower cost and time spent than in-person interviews, with the lowest costs and time associated with virtual-only interviews. Additionally, this study utilized a qualitative analysis of student perceptions of the three interview formats (in-person, virtual, hybrid). We hypothesized that applicants would report greater concern related to the unfamiliarity and uncertainty of the all-virtual and hybrid interviewing process, and that applicants would express a preference for in-person interviewing.

Following a pilot study on a regional campus in 2014-2015,54 all students at the University of Kansas School of Medicine (KUSM) who participate in the NRMP have been surveyed annually since 2015-2016.34,46 The survey questionnaire covers multiple aspects of the NRMP process including the number of applications submitted, interviews offered and completed; estimated costs, including application fees and travel expenses; time invested; and narrative feedback. For the virtual (2020-2021) and hybrid (2021-2022) year surveys, expenses could include costs associated with required technology for virtual interviews such as cameras, microphones, and ring lights. The questionnaire is revised each year based on student feedback, developments in the literature, and changes in the NRMP process and is available upon request from the authors. All NRMP participants receive the questionnaire by email on the day after rank-order list certifications are due and it is accessible until the day before NRMP results are available in mid-March in order to ensure residency match outcomes do not influence respondents. Student leaders send regular reminders encouraging their classmates to complete the survey. As an incentive, a donation proportional to the response rate is available to the class graduation celebration fund. The University of Kansas School of Medicine Institutional Review Board has approved this study as non-human subjects research.

We conducted analyses on data collected from applicants to family medicine programs in the most recent fully in-person interviewing cycle (2019-2020), the virtual-only cycle (2020-2021), and the hybrid interviewing cycle (2021-2022). Descriptive analyses provided demographic information about survey respondents (age and gender). To determine any statistical differences between years, we performed analyses of variance (ANOVA) using IBM SPSS Statistics for Windows, version 26.0 (IBM Corp, Armonk, NY). We used Bonferroni post hoc analyses to adjust for multiple comparisons between variables.

Narrative responses to the question posed in the virtual-only year (2020-2021) and hybrid year (2021-2022) surveys: “How did you personally feel about the virtual format of residency interviewing as compared to in-person interviewing?” were analyzed independently by two investigators (T.R., K.N.) using a thematic analysis approach.55 This method to identify and interpret patterns of meaning across qualitative data follows an inductive process of becoming familiar with the data, generating initial codes, and then classifying and refining shared themes. Both investigators independently coded the narrative comments, reached consensus on an agreed coding framework, then identified common patterns and differing views. The dyad then came to an agreement on the name of each theme, identified explanatory quotations to defend each theme named, and utilized discussion to resolve any differences. Members of the research team, which included students who had completed both in-person and virtual interviews, reviewed thematic findings to help authenticate identified themes.

Data were available for 71 applicants to family medicine programs in all 3 years (overall response rate: 63.4%; 71/112). Yearly response rates were 48.7% (19/39) for the in-person interviewing year, 72.5% (29/40) for the virtual-only year, and 69.7% (23/33) for the hybrid year. Only one respondent in the hybrid year did not complete any in-person interviews. Overall, 47 respondents identified as female (66.2%), and the average age was 26.6 years (±2.1). The age and gender distribution did not differ significantly among years.

We conducted ANOVA to determine differences between the 3 years for all variables. For overall cost, a statistically highly significant difference was found between years (F[2,64]=11.7, P<.0001, χ2=0.2). Post hoc analyses indicated that costs for the in-person interview year were significantly higher ($2,394.70±$1,961.20) than the virtual year ($646.80±$846.60, P=.0001) and the hybrid year ($903.30±$793.40, P=.001). There was no statistically significant difference between the virtual and hybrid years for overall cost. Figure 1a shows the average amount of money spent on the NRMP process per applicant.

For interviewing time, we found a statistically significant difference between years (F[2,66]=15.6, P<0.0001, χ2=0.2). Post hoc analyses indicated that overall time spent interviewing for the in-person interview year were significantly higher (25.9±7.9) than the virtual year (14.9±7.6, P=.0001) and the hybrid year (14.3±7.0, P<.0001). There was no statistically significant difference between the virtual and hybrid years for overall time. Figure 2a shows the average time spent on the NRMP process per applicant.

Additionally, there was a statistically significant difference between years for estimated cost per interview (F[2,62]=12.4, P<.0001, χ2=0.2) and time per interview (F[2,65]=31.3, P<.0001, χ2=0.3). Post hoc analyses indicated that cost spent per in-person interview was significantly higher ($203.80±$160.70) than virtual interviews ($55.90±$83.30, P=.0001) or interviews offered in the hybrid year ($73.50±$44.10, P=.0001). The post hoc analysis also indicated that time spent per in-person interview was significantly higher (2.3±0.7) than virtual interviews (1.2±0.4, P=.0001) or interviews offered in the hybrid year (1.3±0.3, P<.0001). There was no statistically significant difference between the virtual and hybrid years on cost or time spent per interview, or between any of the years for cost and time spent on unranked programs. Figures 1b and 2b show the trend in average amount of money and time spent per interview.

Respondent answers to the question “How did you personally feel about the virtual format of residency interviewing as compared to in-person interviewing?” yielded 74 unique comments in the virtual-only year and 75 unique comments in the hybrid year. We categorized responses with an initial agreement rate of 98.0%. After final discussion, we categorized all responses into five thematic categories. For both years, the categories identified were “feelings related to interviewing,” “suggestions for future,” “convenience/logistics of interviewing,” “perceived fit of program,” and “cost/time of interviewing.” In the virtual-only year, respondents indicated that they had mixed feelings about the interview format, but felt programs did the best they could. They provided suggestions on how to improve the virtual format in the future with many indicating a wish for some in-person contact either through meet and greets or for those already in the programs’ locations. Respondents discussed the convenience and logistics of the virtual format due to the ability to focus on other aspects of the programs, while some were concerned with the lack of understanding of the culture of programs and applicant personal perceived fit in the culture. Seven respondents were pleased with the cost and time savings of virtual interviewing. Table 1 shows the themes with illustrative quotes.

In the hybrid year, respondents also gave suggestions on improving the interviewing process, such as revising the format of interview days and only offering a single format. They discussed a mixture of negative, positive, and neutral feelings toward the hybrid format. Similar to the virtual-only year, respondents also indicated the convenience and logistical considerations of virtual versus in-person interviewing, especially when considering the ability to interview all over the country. Respondents discussed difficulties in assessing fit with programs as an issue with virtual interviewing, and seven indicated the cost and time savings as a benefit. Table 2 shows the themes with illustrative quotes for the hybrid interviewing year.

This study took advantage of a unique, multiyear project tracking NRMP applicant experiences, including financial and time costs, to document the impact of the pandemic-related changes during the all-virtual (2020-2021) and hybrid (2021-2022) cycles. Compared to previous in-person interviewing years, applicants to family medicine programs saved on average $1,748 (73%) during the all-virtual year, and $1,492 (62.3%) during the hybrid year. Although these savings may appear modest for individual applicants, they represent savings of $2.5 million to $3.2 million for the approximately 1,800 US graduates who apply to family medicine programs nationally each year.56 This is especially important as it comes at a time when the rapidly rising indebtedness of family medicine residents is a growing concern. Almost half of family medicine residents reported more than $250,000 of debt in 2019 compared to only 25% reporting the same in 2014.63 Despite the welcome overall savings, the average increase of 40% ($256.50) in the hybrid year and the reported expenditure of $1,500 to $1,700 by individual applicants in the virtual and hybrid years, remain concerning. More could be done to contain costs and prevent increase in future years. In addition to financial savings, students in virtual or hybrid years saved an average of 11 days—precious curricular time that could be used to improve their preparation for transition to resident responsibilities. Data from the growing body of literature surrounding the virtual interviewing process is in line with the current results, revealing the benefits of cost savings, decreased travel time, and reduced environmental impact. 6,64-68

Due to these decreased expenses and easier scheduling associated with virtual interviewing, applicants may apply to and take interview spots in programs that they may not have accepted had it been offered in-person, thus leading to increased numbers of interviews and programs ranked.4,60-62 This could impact the matching algorithm and applicant NRMP success rates; however, it is unknown if this is true for family medicine given the nature of the 2021-2022 hybrid interview season. Further, the role of finances in the application and interview process may have implications in specialty demographics and diversity in medicine.35

Contrary to our expectations, students generally expressed confidence in the virtual interviewing process. One noted concern was difficulty in perceiving subjective aspects of a program (and to a lesser extent the institution and community), including interpersonal communications, relationships, practices, and values in order to assess personal fit. In the hybrid year, respondents made suggestions to consider for future interviewing cycles related to perceptions of their interviewing format choice (ie, “No matter what people will feel obliged to go in-person to increase their chances”) and concerns about equity amongst applicants (ie, that interview format consistency allows for “an even playing filed for all applicants”). In this high-stakes process, if the interview format continues to be hybrid and virtual formats are perceived or rumored to be second best by applicants or programs, the number of applicants opting for in-person interviews may increase, thus re-establishing the cycle of rising costs and time spent on the process. The themes identified in this study are comparable to those in other available studies65-70; however, there are limited studies examining applicant perceptions of the hybrid process as this was unique to family medicine in 2021-2022.

Our study has several limitations, most notably the relatively small number of participants all from a single institution. We were unable to report comparisons by gender and racial/ethnic status without compromising confidentiality of respondents. All data were self-reported and thus vulnerable to recall and other biases. Despite these limitations, our findings clearly demonstrate significant financial and time savings for applicants to family medicine programs associated with virtual or hybrid interviewing formats. We highlight the major concern of enabling applicants to assess personal fit with programs without in-person experience and the danger if virtual interviews are perceived as less desirable or effective than in-person formats.

Virtual interviewing works well from the applicant perspective in terms of overall cost and time, but ways for applicants to get a feel for the culture of residency programs, experience personal interactions with faculty and residents, and identify fit for themselves and their families within communities will need to be identified. One danger with a continued hybrid format is that in-person interviews will be perceived as more desirable/effective or as indicating a greater interest in the program, and virtual will be regarded (even subconsciously) as second best. Issues of equity between those who can afford to attend in-person interviews and those who cannot afford to travel should also be considered. Future investigation of the NRMP interviewing process should include additional qualitative studies to understand motivations and beliefs from both the applicant and residency program perspective.