Use of Signaling in Family Medicine Residency Interviewing

Over the past 2 years, an increasing number of specialties and residency programs have participated in the supplemental application signaling process with the Electronic Residency Application Service (ERAS) in the National Resident Matching Program (NRMP). Signals offer an applicant the opportunity to formally express interest in a specified number of programs.

In the 2023–2024 application cycle, family medicine residency program directors (PDs) will have new information available from applicants regarding their preferences for geographic region in the country as described by census regions and for the setting of the program as described by a continuum of rural to urban setting. Further, PDs will have applicant signal information available when an applicant has indicated heightened interest in a program during the application process. The current study describes family medicine residency PDs’ intended use of signals and provides benchmarking descriptive data that may help inform best practices, modifications to the application process, and future interventional and qualitative studies.

The experience of other specialties with signaling may help predict how family medicine PDs may use signals information in the 2023–2024 application cycle. Otolaryngology (ENT) piloted signaling as a specialty in the 2020–2021 application cycle in an effort to help residency PDs identify interested candidates among a glut of applications for each position. ¹ With multiple application cycles now completed, surveys of ENT participants have suggested that the rate of receiving an interview is higher for applicants in signaled than in nonsignaled programs and that PDs commonly use signals to differentiate between equally qualified applicants when offering limited interview spots. Some ENT PDs even indicated that signals were used as part of determining the rank order list for their matching process. ¹

Additional specialties, such as dermatology, internal medicine, and general surgery, participated in signaling during the 2021–2022 application cycle, noting that signals are an important part of a holistic screening process for each application and may help to determine applicant fit with the program.² Signals were noted to increase applicant odds of receiving an interview or matching to a program when comparing those who did and did not signal in internal medicine.³ Similarly, orthopedic surgery PDs also have noted that applicants who signal a program are more likely to receive an interview at the program than if the applicant does not signal. ⁴

Signaling may be an important addition to multiple methodologies and tools of a holistic review process. Tools for holistic review, such as the ones described by Igarabuza et al, may help minimize the effects of systemic racism and bias that can influence the selection process when deciding which applicants to interview.⁵ Holistic review of applicants has been shown to increase the percentage of women and individuals underrepresented in medicine who interview and matriculate with programs in multiple specialties. ^6-8

The American Association of Medical Colleges defines holistic review as “mission-aligned admissions or selection processes that take into consideration applicants’ experiences, attributes, and academic metrics as well as the value an applicant would contribute to learning, practice, and teaching.”⁹ Through holistic review, each applicant is viewed as a complex individual rather than a simplified series of quantifiable metrics. With high application volumes, PDs may have used scores, ranks, or admissions to honor societies as a means to select applicants for interview. Individuals who are traditionally underrepresented in medicine have been shown to often receive lower grades, have poorer clinical evaluations, be less likely to be admitted into the Alpha Omega Alpha honor society, and perform poorer on standardized tests due to structural bias.^10-12 Further, measures such as Alpha Omega Alpha status, class rank, or United States Medical Licensing Exams performance do not correlate with residency performance; thus eliminating applicants on the basis of such measures will not result in a better resident class.¹³ Signals allow PDs to use student interest to screen applicants and minimize the use of other metrics such as standardized examination scores or class rank.

The survey questions used for this study were part of a larger omnibus survey conducted by the Council of Academic Family Medicine Educational Research Alliance (CERA). The methodology of the CERA Program Director Survey has previously been described in detail.¹⁴ The CERA Steering Committee evaluated questions for consistency with the overall subproject aim, readability, and existing evidence of reliability and validity. Pretesting was done on family medicine educators who were not part of the target population. Following pretesting, questions were modified for flow, timing, and readability. The project was approved by the American Academy of Family Physicians Institutional Review Board in April 2023. Data was collected from April 18 to May 12, 2023.

All Accreditation Council for Graduate Medical Education (ACGME) accredited US family medicine residency PDs, as identified by the Association of Family Medicine Residency Directors, were invited to participate in the CERA omnibus survey. Email invitations to participate from the CERA staff were delivered with the survey using the online program SurveyMonkey (SurveyMonkey Inc). After the initial email invitation, three follow-up emails to encourage nonrespondents to participate were sent weekly, and a fourth reminder was sent 1 day before the survey closed. PDs totaled 745 at the time of the survey. PDs with undeliverable email addresses were excluded from the omnibus survey. Because most CERA topics require some knowledge of all aspects of residency, the CERA methodology does not permit any PD whose program has not yet graduated a class of residents and had three resident classes who have experienced all aspects of training to complete any of the survey questions. CERA sets the core, recurring demographic questions asked for all PD surveys, while we (the authors) crafted the 10 module questions.

Individual survey questions were devised to test a series of 10 hypotheses related to PDs and their use of signals in the residency interviewing process, as described here.

We hypothesized that PDs are more likely to use signals to decide which applications to review and who to invite to interview than to rely on who to rank or where to rank an individual on the rank order list.

Further, we hypothesized that program characteristics such as geographical location, size, or type would not be associated with a PD’s decision to use applicant signals. Similarly, we hypothesized that PDs would recommend applicants who signal residency programs of most interest regardless of prior clinical interactions or program affiliation. We further hypothesized that PDs would agree that signals would decrease stress on applicants, decrease stress on PDs, make the NRMP process more equitable, and decrease the ratio of interviews completed per intern position. However, we also hypothesized that PDs additionally would be likely to indicate that signals would not change the number of residency applications submitted. Finally, we supposed that the completion of a rotation with the program would be the most likely factor to act as a tiebreaker if offering applicants an interview or determining an applicant rank.

We used descriptive analyses (numbers and percentages) to describe participants’ demographics (gender, degree, race/ethnicity, and PD experience), as well as their residency program demographics, such as program size, program region, program type, and community size. We used ꭓ² and Pearson correlation analyses to examine any association between how family medicine residency programs intend to use signaling and their perceived impact of signaling on the residency interviewing process. We used a multivariate general linear regression analysis to assess how program demographics (residency region, residency size, and residency type) impacted participants’ use of signals, with a P value of .05 as statistically significant. Statistics were completed using SPSS version 29.0 (IBM).

Although 745 PDs were identified by the Association of Family Medicine Residency Directors at the time of the survey, 48 were excluded, reducing the sample size to 691. Six PDs were excluded due to undeliverable email addresses, while 42 were removed because the program had not yet graduated a class of residents.

The overall response rate for the survey was 44.72% (309/691). Approximately half (52.4%; 162/309) were women and 77.0% (238/309) were doctors of medicine (MDs). We reviewed the publicly available family medicine residency PD listing from the ACGME website to determine whether our respondents appropriately represented the demographics of PDs. We noted that our sample included a slightly higher percentage of women (52.4% vs 46.7% nationally) and a slightly higher percentage of MD graduates (77.0% vs 74.8%).¹⁵ Similarly, respondents had been in their PD position for an average of 5.8 years (±5.3). This compares favorably with data reported by Brown and Gerkin in 2019, which indicated that the average family medicine PD tenure was 6.5 years and the median tenure was 4.5 years.¹⁶ Further, the majority of respondents identified as White (68.6%; 212/309), which is similar to what Weidner and Clements found (71%) when reviewing PD race and ethnicity in 2021. ¹⁷

With respect to the programs represented, more than half of respondents’ residency programs were community-based, university-affiliated (55.3%; 171/309) and 33.0% (102/309) were in the southern region of the United States. Nationally, 54.6% of family medicine residency programs are community-based, university-affiliated, and 33.0% are in the southern region of the United States. Most programs had 19 to 31 residents (43.0%; 133/309) and were in communities smaller than 500,000 people (68.2%; 211/309). Table 1 gives full demographic information for respondents and their residency programs as well as comparisons to national data, where available.

PDs are more likely to not use signaling when deciding which applicants to review in-depth (ꭓ²[1]=44.3, P<.0001, 95% CI 19.0% to 34.1%), which to rank (ꭓ²[1]=298.9, P<.0001, 95% CI 63.3% to 74.7%), or where to rank individual applicants (ꭓ²[1]=231.1, P<.0001, 95% CI 54.4% to 66.9%). However, PDs were more likely to use signals to decide which applicants to invite for interview (ꭓ²[1]=28.8, P<.0001, 95% CI 13.8% to 29.1%). Many PDs would use a signal as a positive factor in a holistic review of the candidate (66.7%, 206/309; Appendix Table A).

PDs who indicated they would use signals to decide which applicants to review in-depth were more likely to also use signals to decide which applicants to invite to interview (r= 0.127, P=.03). Those who indicated they would use signals to decide which applicants to invite to interview also were more likely to use signals when deciding which applicants to rank (r=0.211, P<.0001); and PDs who indicated they would use signals when deciding which applicants to rank were more likely to use signals when deciding where to rank individual applicants (r=0.45, P<.0001; Appendix Table A).

We carried out a multivariate general linear regression analysis to assess the effect of residency region, residency size, and residency type on the likelihood of PDs’ using signals to decide which applicants to review in-depth, using signals to decide which applicants to rank, using signals to decide where to rank individual applicants, or choosing not to use signals in their decision-making. We found no statistically significant difference between residency region, size, or type (R²=0.5, F[44, 966]=1.1, P=.3).

When faced with equivalent applicants, 34.6% of PDs indicated that an applicant completing a rotation with their program (107/309) would be the factor most likely to sway them to offer an interview. Similarly, 23.6% of PDs (73/309) indicated that an applicant being from the same state as their program and/or likely to practice in the state of their program would be the most likely factor to sway them to offer an interview. Only 9.4% of PDs (29/309) selected applicant signaling interest through ERAS as the most likely factor to sway them to offer an interview.

Similarly, when determining which factor would be most likely to sway a PD to rank one equivalent applicant higher than another 34.6% (107/309) selected completing a rotation with the program, 18.1% (56/309) selected being from the same state where the program is located and only 4.9% (15/309) selected applicant signaling interest through ERAS as the most important factor. Furthermore, PDs did not indicate a belief that signals would decrease stress on either applicants or themselves. Rather, many PDs were neutral when asked if signals will decrease stress on applicants (50.5%; 156/309) and a plurality were neutral asked if signals will decrease stress on PDs (41.7%; 129/309; Appendix Table A).

When considering the impact signals may have on the NRMP process, most PDs (56.3%; 174/309) indicated a neutral stance when asked whether signals will make the NRMP process more equitable. Similarly, most PDs responded that signals would not change the number of applications submitted (70.9%; 219/309) and would not change the ratio of interviews completed per intern position (78.6%; 243/309). Most PDs responded that they would recommend that applicants signal residency programs in which they have the most interest regardless of prior clinical interactions or affiliation with the program (77.0%; 238/309; Table 2).

Signaling has been proposed as an intervention that may improve the ability of PDs to quickly identify those applicants most likely to be a fit for their program and thereby may prove beneficial with the hundreds of applications a PD is asked to review each season. Most of the PDs in this study indicated that they did not anticipate signaling to result in fewer applications, fewer interviews, or a reduction of stress for applicants or themselves. Interestingly, however, preliminary data released by the American Association of Medical Colleges may suggest otherwise, at least with respect to application numbers. In the 2024 match, individuals applied to an average of 48.72 family medicine programs, which is a decrease from the 56.39 average noted in the 2023 cycle.¹⁸ Similarly, an individual program received an average of 604.82 applications in 2024, which decreased from 758.76 in 2023.¹⁸ The impact of signaling on interviews and stress levels is not yet known, but should be assessed following the conclusion of the 2024 cycle.

Although signals may or may not reduce the workload or stress typically associated with interview season, the current results suggest that an applicant would be unwise to disregard the importance of signals in the process. Indicating or failing to indicate interest in a program with a signal may be a critical decision that impacts an applicant throughout the NRMP season. In other specialties, signals have been observed to be a key component in determining who does and does not receive an interview with a specific residency program. ^{1, 2}

Currently, many family medicine PDs anticipate using signals as part of a holistic review and in deciding which applicants to invite for an interview. When used in this context, signals may be one way in which applicants may distinguish themselves. Additionally, the move toward a more holistic review process may have important implications for removing bias that may have impeded efforts to encourage diversity within programs. In some specialties, signals may be a quantifiable value that allows for rapid sorting of applications at a time when other quantifiable values such as Step 1 scores are disappearing. However, in family medicine, a single factor such as a signal may not be as important as reliance on qualitative factors as part of a holistic review. A signal may be an important factor in holistic review but may not be the most, or only, important factor in a tie between two equivalent applicants. The current results suggest that other factors, such as completing a rotation with the program or intending to practice in the state where the program is located, may carry more weight than a signal.

PDs in family medicine will experience signals for the first time as part of the 2024 interview season. Answers to this survey may reflect a theoretical intention to use signals that may not align with the behavior that will be observed when signals are implemented. Reducing application volume to programs, improving alignment of applicant and programs applied to, and reducing stress for both PDs and applicants throughout the process are goals that many in medical education may support; however, the path to achieve these aims is not yet clear. Signals may play a critical part in moving toward a new recruitment paradigm for graduate medical education; however, the impact of signals in family medicine has yet to be determined.

This study has recognized limitations. While the survey was sent to all family medicine PDs, only 44.72% responded. The respondents appear to approximate the general family medicine PD population with respect to gender, degree, type of residency program, region of residency program, and size of program. Information was not available to determine whether the respondents to the survey were representative of the PD population as a whole with respect to race, ethnicity, identification as underrepresented in medicine, or experience as a PD. Furthermore, because the methodology for the larger omnibus survey excludes PDs who have not yet graduated a class of residents, those individuals were excluded from our data. Whether their perceptions about signaling would have varied from those presented by the respondents is unknown.

Moreover, this study assessed only PDs’ anticipated use of signals and did not address how medical students may use signals, nor did it investigate how medical school faculty may advise medical students given the addition of signals to the application process. Future research should build on the current results to investigate observed patterns of interaction with an aim to improve the recruitment and interview experience for all parties involved.