New Tools for Resident Recruiting Season: A CERA Survey of Family Medicine Residency Program Directors

Residency recruitment strategies continue to evolve after two cycles in which virtual interviewing models were strongly recommended by governing bodies. While the use of virtual interviewing models has been proven a successful option, the convenience of the model has led to concerns regarding applicants’ overapplying and overinterviewing for residency positions. Electronic Residency Application Service (ERAS) data has shown a continued upward trend in the total number of applications per applicant for family medicine programs. ¹ This trend persists despite the most recent match data from the National Resident Matching Program (NRMP) showing that US medical school seniors (MDs & DOs) have approximately 90% match success when they rank 5 to 6 programs and near 100% success when they rank 12 to 13 programs. After 13, the applicant has no increased benefit from additional program ranking. ^{2, 3}

Additional interview season strategies that have been suggested or trialed since the start of the pandemic in other specialties include capping the number of residency interviews an applicant can accept, ⁴ along with preference signaling, where an applicant has the ability to communicate to a select few programs a more sincere interest in training there. ^{5, 6} A supplemental application within the ERAS system also has been used and has proliferated among multiple specialties over recent interviewing cycles. ^7-9 While these formalized models have yet to be trialed in family medicine residency programs, ongoing discussions are taking place among stakeholders as to the potential use of these tools as early as the 2023-2024 residency recruiting season. ¹⁰ Of particular interest are the use of a supplementary application and preference signaling.

In 2021, the Council of Academic Family Medicine (CAFM) Educational Research Alliance (CERA) fielded a survey module to family medicine program directors focusing on the use of supplemental applications, preference signaling, and interview capping as standardized components of the specialty’s recruiting season. Data from the survey provide initial insights into residency program directors’ perceptions of these issues. The purpose of our study was to measure program directors’ perspectives on these potential new components to resident interviewing and to assess for variables, including demographics, that may influence program directors’ opinions on these topics.

CAFM is a leadership and research collaborative among the leading academic family medicine organizations: the Society of Teachers of Family Medicine, the Association of Departments of Family Medicine, the Association of Family Medicine Residency Directors (AFMRD), and the North American Primary Care Research Group. CERA’s program director survey methodology has previously been described in detail. ¹¹ Questions were modified based on feedback received. We were given ethical approval for the study by the American Academy of Family Physicians Institutional Review Board in September 2021. The survey was fielded from September 14 to October 8, 2021, and we conducted data analysis November 2021 through March 2022.

The directors of all US family medicine residency programs accredited by the Accreditation for Graduate Medical Education, as identified by the AFMRD, served as the survey sampling frame. SurveyMonkey (Momentive Inc) was used to deliver email invitations and reminders to participants and collect survey responses. Three weekly follow-up emails were sent to nonrespondent participants, and a fourth reminder was sent 2 days before the survey closed. Of the 696 program directors on the distribution list, 30 emails returned a bounced message. Programs with fewer than three resident classes were excluded (n=35) because those programs have yet to experience a graduating class of residents. The final sample size was 631.

We obtained residency program demographic data from the recurring questions of the survey. We combined the underrepresented in medicine (UriM) residency percentage (0%) group with the less than 6% group because very few in the sample reported 0%. Additionally, we dichotomized support for each topic by grouping “Strongly agree” and “Agree” responses to the survey question.

The frequency of responses and percentages are reported for categorical variables, and mean and standard deviation are reported for continuous variables. We performed bivariate analysis among the three module questions to determine significant, either statistically or of high magnitude, associations. For the variables with a significant association, we conducted multivariate logistic regression to examine impacts adjusted for demographic factors. We used Stata version 14.0 (StataCorp) to perform the statistical analyses, with a P value of.05 as statistically significant.

The overall response rate to the survey was 44% (275/631). The response rate for our module questions was 42% (n=262). In the program directors’ module, certain demographic responses were particularly relevant to experience with virtual interviewing: the mean years in their current position (5.6); a minority (8%) identifying as Hispanic; and the majority of programs being community-based, university affiliated (58%, see Appendix A).

We found strong support for capping the number of applications per applicant and for applicant preference signaling. A significant majority of program directors were in favor of a national interview cap and preference signaling (both 68%, Table 1). In contrast, only 44% of respondents were in favor of making a supplemental application a standard component of the recruitment process.

We found a statistically significant relationship between favor for preference signaling and favor for supplemental applications. For these two questions, we combined those who responded “neither agree nor disagree” with those who responded “somewhat disagree” or “strongly disagree” to create a “not agree” category. More program director respondents who favored preference signaling also favored supplemental applications than program director respondents who did not favor preference signaling (52% vs 29%, Table 2). Overall, 241 program director respondents provided complete demographic information and were included in the multivariate logistic regression. When adjusting for residency demographic factors, the program directors who supported supplemental applications were three times more likely to favor preference signaling (odds ratio [OR]: 3.16; 95% confidence interval [CI]: 1.51-6.60). Compared to program director respondents whose residency was located in communities with less than 30,000 residents, all other program director respondents were less supportive of preference signaling, ranging from 81% to 88% less likely (all significant at the 10% level).

Compared to program director respondents whose residency consisted of less than 19 residents, program director respondents with a larger residency size were less likely to favor preference signaling (19-31 OR: 0.47; 95% CI: 0.21-1.01; more than 31 OR: 0.57; 95% CI: 0.19-1.74). Additionally, compared to female program director respondents, male program director respondents were less likely to favor preference signaling (OR: 0.36; 95% CI: 0.17-0.76).

This CERA survey’s responses provided early insight into the perspective of family medicine program directors for whom new tools would be helpful to counteract some of the challenges of the ever-evolving residency recruitment season. Supplemental applications have been used by family medicine residency programs in the past, but at a program level, and not necessarily in direct coordination with ERAS. At the time this survey was collecting data in September to October 2021, only otolaryngology residencies had completed an interview season using preference signaling in their recruiting; dermatology, internal medicine, and surgery were starting their first experiences during that time frame. ¹¹

Because some questions have been raised about the legality of a national interview cap as it pertains to trade laws, ¹² the use of supplemental applications and preference signaling would seem to hold the highest potential for family medicine residency programs in the upcoming match seasons.

Our statistical analyses showed program directors generally supporting both supplemental applications and preference signaling, as opposed to supporting one tool independently. With specific regard to preference signaling, program directors in smaller communities or with smaller class sizes showed greater support for this system; that finding could be due to these program directors’ desire to use preference signaling as a tool to refocus their recruiting resources on applicants with particular interest in training at their site. The most relevant data on preference signaling came from the 2021 survey of program directors’ and applicants’ experiences in otolaryngology, where program directors found it most helpful when a preference was expressed from an applicant when it was not otherwise evident (eg, from a different geographical region). ¹³, ¹⁴ Further investigation of this relationship would be beneficial, particularly to programs in smaller or rural communities. Additionally, the preference difference we found between male and female program directors may simply be an incidental finding and requires further study.

One potential limitation of this study was the simplistic framing of the questions, without specific descriptions of what each question’s subject would entail. This limitation was especially true for the questions involving a national interview cap and supplemental application; the latter particularly would have the potential to change responses based on a more detailed overview of what the application would require.

In conclusion, family medicine program directors showed support for use of all three proposed recruiting season tools, with preference signaling and interview capping receiving the greatest backing, and support for all tools more likely than isolated support for one tool. When these responses were further analyzed, program director gender, residency community size, and residency class size were variables that significantly affected support.