Financial Considerations After a Fully-Virtual Interview Season: A CERA Study of Family Medicine Residency Program Directors

Residency programs in the United States have faced unprecedented challenges since the beginning of the COVID-19 pandemic, including those affecting residency recruitment seasons. Traditionally conducted via in-person interviews with candidates at residency sites, the vast majority of family medicine residency programs were forced to conduct a fully-virtual interview season during the 2020-2021 recruitment season. This drastic transition to a system devoid of in-person interaction would have been unthinkable to most program directors (PDs) prior to the onset of restrictions related to the pandemic. However, many program directors reported the pleasant surprise of a net-positive experience during the 2020-2021 virtual interview season.1 Questions remain as to how residency programs will proceed as COVID-19 restrictions are relaxed and lifted, as well as the ultimate success of the initial 2020-2021 virtual interview season; since meeting applicants in person and attempting to determine their fit in a program has traditionally been a hallmark of the residency recruitment process.

Conversely, with the aforementioned positive reviews from the initial foray into virtual interviewing, and an additional year of preparation and refinement of these processes in 2021-2022, advisory bodies have seen an opportunity for this paradigm shift to better serve applicants and promote equity. In spring 2021 the Coalition for Physician Accountability published Initial Summary Report and Preliminary Recommendations of the Undergraduate Medical Education to Graduate Medical Education Review Committee, which included recommendations for long-term changes to the residency recruitment process.2 Among the recommendations were a fully-virtual interview season for 2021-2022 (recommendation #26) with “ongoing study of the impact and benefits of virtual interviewing as a permanent means of interviewing for residency.”

While the Coalition paper framed the debate of in-person versus virtual interviewing primarily from the pro/con perspective of the applicant,2 particularly highlighting the financial benefits to an applicant interviewing in a fully-virtual model, the perspectives of residency programs and their leadership team should also be considered before any permanent changes are made to the recruitment process. Residency recruitment is well known to encompass a significant percentage of programs’ annual budgets; a previous CERA study put the average per-applicant amount spent at $213, with a very wide confidence interval (CI), meaning costs often are considerably higher.3 Thus, there are similar financial benefits to residency programs to a virtual-centric interview season, and assessing program director perceptions remains an opportunity to gauge perspective that can guide modeling of future interview seasons.

To begin the investigation into residency program directors’ perceptions of virtual interviewing, in 2021 the Council of Academic Family Medicine (CAFM) Educational Research Alliance (CERA) chose our module on the financial impacts and preferences regarding the 2020-2021 interviewing season for their Family Medicine Residency Directors Survey. The survey was subsequently fielded to all Accreditation Council for Graduate Medical Education- (ACGME) accredited US family medicine residency program directors. The purpose of this study was to (1) measure perspectives of family medicine residency program directors pertaining to virtual interviewing and the means by which future interview seasons will proceed, (2) identify potential financial factors that lead program directors to favor virtual or in-person interviewing models, and (3) assess PDs perceptions about whether the ameliorating effect of a virtual interview season’s financial impact on residency applicants supersedes the benefits of in-person interviewing.

CAFM is a leadership and research collaborative between the Association of Departments of Family Medicine, the Association of Family Medicine Residency Directors (AFMRD), the North American Primary Care Research Group, and the Society of Teachers of Family Medicine. The methodology of the CERA Program Director Survey has previously been described in detail.4 Questions from the five selected topics were evaluated for consistency with the overall subproject aim, readability, and existing evidence of reliability and validity by the CERA Steering Committee. Family medicine educators who were not part of the target population pretested the survey for flow, timing, and readability. We modified questions based on feedback received. The American Academy of Family Physicians Institutional Review Board gave ethical approval for this study in September 2021. Data were collected from September 14, 2021 to October 8, 2021 and we conducted data analysis in November 2021 through March 2022.

The survey sampling frame was all ACGME-accredited US family medicine residency program directors as identified by the AFMRD. We used the online program SurveyMonkey to deliver email invitations to participants and collect survey responses. Three follow-up emails to nonrespondent participants were sent weekly after the initial email invitation and a fourth reminder was sent 2 days before the survey closed. The survey was emailed to 696 program directors. A total of 666 invitation emails were delivered as 30 email addresses returned a bounced message. The survey began with an exclusion question to remove programs with fewer than three resident classes. The final sample size was 631 as 35 program directors indicated fewer than three resident classes.

We obtained demographic data from the recurring questions of the survey (Appendix A). For our analyses we combined the underrepresented in medicine (URiM) residency percentage categories, the 0% group with the less-than-6% group as there were very few in the sample reporting 0%. We also dichotomized support for a fully-virtual interviewing session by grouping strongly agree and Agree responses to the question, “I support moving to a fully-virtual interviewing model for resident recruitment due to the cost-savings over in-person interviewing.”

We generated descriptive statistics for all variables. The total number of responses and percentages are reported for categorical variables and mean and standard deviation are reported for continuous variables. We performed bivariate analysis between the module questions and the demographic variables to determine significant—either statistically or of high magnitude—associations, and we used multivariate logistic regression to examine adjusted impacts for significant associations. We performed statistical analyses using Stata 14.0 (StataCorp, College Station, TX) and we used a P value of.05 to determine statistical significance.

The survey received 275 respondents for an overall response rate of 43.58% (275/631). After the PD demographic questions, the response rate declined to 263 for the five sections with substantive questions, for an effective response rate of 41.7%. As with previous CERA surveys, the PD respondents were similar to the universe of PDs (see Appendix). Of particular relevance to experience with virtual interviewing, the mean years in their current PD position was 5.6; a minority (7.6%) identified as Hispanic, and the majority of programs were community based, university affiliated (58.2%).

The majority (91.3%) of responding programs implemented a 100% virtual 2020-2021 interview season (Table 1). Additionally, 38.8% of residency program directors agreed with the statement, “I support moving to a fully virtual interviewing model for resident recruitment due to the cost-savings over in-person interviewing,” and 17.6% were neutral. Almost one-third (30.8%) of responding program directors reported not knowing if their funds recouped were accessible for other residency operating costs; 32.3% reported yes, and approximately 36.9% reported no. Over half of PDs (59.3%) reported their program offered meal costs to residents prior to the 2020-2021 interview year, 49.8% offered lodging costs, 13.3% offered other financial incentives, and only 1.5% covered flight expenses.

We found a wide variation in PDs’ attitudes about future use of virtual interviews. Although 86.6% of respondents stated they would follow recommendations for a 100% virtual 2021-22 interview season, 30.4% prefer fully virtual, 22.4% prefer in-person, and 33.8% prefer hybrid virtual/in-person interviewing. Overall 66.5% of residency program directors agreed their residency program intends to maintain some capacity for virtual interviewing beyond the 2021-22 interview season. Only 35.7% of responding PDs agreed with the statement, “Applicants benefitting from less financial burden during a fully-virtual interview season supersedes a residency program’s benefit from meeting and assessing applicants in-person (prior to deciding if and where to rank them for the Match).” Twenty-eight percent were neutral, and 35% disagreed.

Bivariate Pearson χ² analysis revealed a significant association between supporting a fully-virtual interviewing model due to cost savings and recouping interviewing funds (P=.000). After controlling for demographic factors, multivariate logistic regression showed PDs who are able to recoup their recruitment budget funds are 3.94 times (CI: 1.69-9.18) more likely to support moving to a fully-virtual interviewing model than those who are not able to recoup funds. The only statistically significant finding among analysis of all survey demographic information included within the CERA omnibus survey bundle, was Hispanic ethnicity of the program director. Compared to PDs identifying as Hispanic, residency programs who identified as non-Hispanic were 8.24 times (CI: 1.2-56.1) more likely to support fully-virtual interviewing.

There are numerous factors to consider when calculating the overall efficacy of virtual interviewing, particularly in comparison to the conventional system of in-person interviewing utilized by programs for decades. The financial aspect of resident recruitment, and the role of virtual interviewing on residency program finances, holds enough significance to programs and departments to warrant additional reflection on whether there will be a role for virtual interviewing in a postpandemic graduate medical education environment. A survey of both applicants and program directors, conducted immediately after the 2021 Match by the National Residency Matching Program (NRMP), showed both the applicants’ perceived importance of reduction of financial constraints on interview-related travel (85.0% indicating slightly important or above), as well as the PDs’ perceived benefit of the reduced applicant-related hosting duties (72.1%).5

Previous CERA data from the 2020 Program Directors Survey further established that (1) recruiting costs to the program are a factor in offering virtual interviews for a majority of respondents (59.4%), and (2) a large majority (83.0%) of respondents’ residency programs’ recruiting costs would decrease through the participation in a virtual interviewing season.6 We found these costs took the form of lodging (63.3%) and meals (75.4%) for the majority of respondents.

When explicitly asked whether long-term resident recruiting should remain fully virtual due to the cost savings over in-person interviewing, there was a wide distribution of responses across all options without consensus. This was supported in the statistical analysis of the data across multiple variables: financial incentivization does not statistically impact support for virtual interviewing overall. While cost savings and in-person applicant assessment hold strong value to their respective invested parties, there was not a statistical significance showing importance of one over the other in our data set.

One of the main benefits of virtual interviewing to applicants, identified within the Initial Summary Report and Preliminary Recommendations of the Undergraduate Medical Education to Graduate Medical Education Review Committee, was its being a “phenomenal change to control applicant expenses.”2 However, this level of enthusiasm was not echoed by all CERA survey respondents: the question asking if applicants’ decreased financial burden was a more important factor than a residency program’s ability to assess candidates in-person yielded a near-even split in answers with regards to which factor was more important to the overall process.

However, in some areas financial considerations did yield a significant influence over PDs’ opinions on the topic of virtual interviewing. PDs who knew that they would be able to recoup the funds earmarked for recruiting season did exhibit a statistically significant support of fully-virtual interview seasons moving forward. This finding seems to indicate that these PDs find virtual interviewing to be a worthwhile system, with the added benefit of cost savings from the traditional in-person interviewing model; or at least the program directors do not feel any negative aspects of virtual interviewing supersede the benefit of additional funds for their residency budgets.

Additionally, the drivers of the significant impact of Hispanic ethnicity of the PDs are uncertain and deserve further investigation. It is possible that Hispanic PDs are more likely to be located in underserved locations and want to showcase their outreach and the diversity of their program through in-person interviewing to generate full appreciation for their residency program. Although the residency community size variables were not statistically significant in our model, some underserved areas are located in larger communities. Thus, future studies are needed to explore these effects in greater depth.

In conclusion, the significant majority of family medicine PDs have embraced virtual interviewing as a recruiting tool for the present and the future, and recognized the financial value of virtual interviewing from all stakeholders’ perspectives, with financial considerations generally not playing a significant role in this decision. There are few, specific examples where financial considerations do influence the preference for future virtual interviewing utilization, including PDs who are able to recoup their recruitment budget funds and Hispanic ethnicity of the PD.