The Influence of the Residency Interview Format on Future Interviewing Models and Use of Preference Signals

The COVID-19 pandemic forced major changes to residency recruitment models across graduate medical education. Residency and fellowship programs, including those within family medicine, quickly pivoted to virtual interviewing models during the 2020–2021 recruitment season, necessitated by continued restrictions of in-person events throughout society in the early months of the pandemic. In the subsequent years (and interview seasons), family medicine residency programs have wrestled annually to decide whether their interview processes would remain virtual or return to the traditional in-person model. ^1-4

An additional component of the residency recruitment process, developed partly by the impact of virtual interviewing models on application numbers, has been preference signaling, where an applicant can communicate a more sincere interest in a select number of programs within the Electronic Residency Application Service (ERAS). This tool, first piloted by otolaryngology during the 2021 interview season ⁵ and available to family medicine residency programs during the 2023–2024 interview season, was received with mixed expectations. ⁶

After one recruiting season with preference signals in family medicine, our research team had the opportunity not only to revisit the current perspective of family medicine program directors on the future of their interview season structures, but also to analyze this and other factors’ effect on the future use of preference signals. Our primary hypothesis posited that a significant proportion of program directors found preference signaling valuable during the 2023–2024 interview season. We also hypothesized that program directors would indicate that five or fewer signals are appropriate for applicants and that no difference exists in support for preference signaling between programs that conduct in-person or hybrid interviews and those that remain fully virtual.

The Council of Academic Family Medicine Educational Research Alliance Program Director Survey methodology has been described in detail. ⁷ Our work group devised 10 questions on family medicine residency program directors’ perceptions of interviewing format and preference signaling for the spring 2024 survey iteration. The project was approved by the American Academy of Family Physicians Institutional Review Board in April 2024. Data were collected from April 30 to June 7, 2024.

We analyzed data using cross-tabulation and χ² tests to assess the relationship between interview formats (virtual vs in-person or hybrid) and the use of preference signaling. We subsequently performed multivariable logistic regression analyses to quantify the effects of interview format, residency program type, and demographic characteristics on outcomes.

For our regression analyses, we used multivariable logistic models to evaluate the likelihood of agreement with maintaining virtual interviews and the number of preference signals considered appropriate. We reported regression coefficients, odds ratios, and 95% confidence intervals; statistical significance was determined at P<.05. All statistical analyses were performed using Stata (StataCorp) software.

Survey data were collected from 382 respondents; 320 of the 382 responded fully to the survey, for an overall response rate of 45.4% (320/705).

Preliminary response analysis identified significant associations between interview format selection and two key outcomes: respondents’ willingness to maintain some capacity for virtual interviews in the future, and their preferences regarding the number of preference signals deemed appropriate for family medicine residency applicants.

Programs using a 100% virtual interviewing format agreed at a statistically significant level that they plan to continue using this model in the future (Table 1); multivariable logistic regression (Table 2) further confirmed this finding. In terms of demographic variables, the only statistically significant difference in whether programs intend to use virtual interviews going forward is whether they currently use that interview format; all the other factors were not significant. Program director gender and ethnicity also did not significantly impact agreement. These findings (Table 1 and Table 2) underscore that support for continuing virtual formats is not influenced significantly by demographic factors, but rather by the currently used format.

Regarding preference signaling, most respondents across all formats favored five signals (Table 3 and Table 4). This result suggests a consensus among program directors on a moderate number of signals for effective communication of interest; but no demographic variables tested identified statistically significant relationships to preferred number of preference signals. A statistically insignificant but noteworthy increase in support for decreasing preference signal allotment down to 0 or 1 was noted in the 100% in-person interview cohort (N=34, 24%) when compared to 100% virtual (8%) or hybrid (14%) programs.

Our study revealed a strong association between the structure of the 2023–2024 interview season and program directors’ intentions for future interview formats. Programs that have continued to use a virtual interviewing model were significantly more likely to express a desire to maintain some capacity for virtual interviewing in the future. This finding aligns with previous research indicating long-term support of virtual interviews in residency recruitment.⁸ Concurrently, a growing number of programs have reinstituted in-person options into their recruiting season, for reasons previously identified in literature, ² including evaluating subjective aspects of a program (“fit” with the people and within the community).

Additionally, the 2023–2024 interview season structure preference was significantly associated with the desired number of preference signals: programs overall, especially those with virtual components in their recruitment process, favored the current five signal allotment. This finding could indicate that program directors who use virtual interviewing formats more often report value from the signaling tool for gauging applicant interest and managing the larger applicant pools often associated with virtual recruitment.^{9, 10} Furthermore, programs reinstituting in-person interviewing may see this model in itself as a signal, particularly in the age of virtual interviewing; if an applicant is committing personal resources to visit a program, this is often seen as signaling particular interest in a residency. Further investigation into whether signals hold the same value to programs trending back toward in-person interviews would be of potential interest. Additional potential areas of study could include more thoroughly evaluating the future interviewing planning in those residencies that currently perform hybrid or in-person interviews, as well as investigating interview format’s effect on applicant behavior and program preferences.

While this study had limitations, the information gained from this analysis provides insight into the bigger picture of how technology-dependent systems are affecting health care, including residency recruitment. Today’s family medicine residency applicants will be the future physicians, leaders, and policymakers of the health care community. Their experiences in these virtual models have the potential to shape practice for decades to come. ¹¹

Miranda A. Moore reports receiving funding from Agency for Healthcare Research and Quality, Health Resources and Services Administration, Ardmore Institute of Health, Alzheimer’s Association, and Georgia Department of Human Services.