Signals and Preferences: Experiences of Midwest Family Medicine Residencies

In the context of significant recent increases in application volume and burden on residency programs, the Association of American Medical Colleges (AAMC) introduced program signals to allow residency applicants to express genuine interest in training programs. ^1-3 Geographic and setting preferences allow applicants to share up to three of nine U.S. census divisions and rural, suburban, and urban settings with free-text narrative explanations of their selections. ⁴ Program signaling was first used by otolaryngology during the 2020–2021 residency application cycle ² and expanded to a total of 22 specialties in the 2022–2023 season. Studies evaluating the impact of program signaling in other specialties have identified an association with decrease in application volume, increased interview offers, ^{5, 6} and largely positive experiences from applicants and residency programs related to incorporating signaling into holistic review to identify residents who may be a good fit for a program. ⁷

Family medicine adopted program signaling in the most recent 2023–2024 application cycle and allows applicants five signals, similar to pediatrics (5 signals) and internal medicine (7 signals in 2023–2024). Family medicine is the largest and most widely distributed primary care specialty in the United States, with 92% of residency program graduates practicing primary care. ⁸ While family medicine residency programs and training slots are expanding to address a shortage of primary care physicians crisis, programs are challenged by increases in unfilled residency positions in comparison to higher paying medical specialties. ⁹ Because the majority of family physicians work within 100 miles of their residency training site ¹⁰ and serve the diverse needs of underserved populations and communities, the unique impact of program signaling and geographic preferences on family medicine training outcomes may differ from other specialties. A preimplementation study of family medicine program directors indicated that program signals were perceived as a positive factor in holistic review and that geographic ties were also important in selecting interview candidates. ¹¹ While signals and preferences are promising evidence-based interventions to assist programs in selecting appropriate residency candidates, little is known about family medicine program experiences, specifically how and why they chose their implementation strategy and the impact on other outcomes of residency training, such as diversity and community health.

To address this gap, we performed a qualitative study to understand challenges and identify and share best practices. Our primary aim was to describe program experiences with implementation of signaling and geographic and setting preferences. Our secondary aim was to identify opportunities for applicants, advisors, programs, and policymakers to optimize the use of signals and preferences.

This was a qualitative study of program directors and other faculty responsible for recruitment at Midwest U.S. family medicine residency programs. The study was informed by the RE-AIM (reach, effectiveness, adoption, implementation, and maintenance) framework. We conceptualized program signaling and geographic and setting preferences as evidence-based innovations from other specialties, now being implemented in family medicine. We selected questions from each RE-AIM domain relevant to our research question: reach, to understand the target population signaling might help; effectiveness, to understand their perceived outcomes of implementation; adoption, to understand how they incorporated signaling and preferences; and maintenance, to derive recommendations for modifications. This study was deemed not regulated by the Institutional Review Board of the University of Michigan (HUM00243040) and Rush University (Non-Human Subjects 118). The description of this study followed the consolidated criteria for reporting qualitative research. ¹²

The study originated within the Family Medicine Midwest Foundation (FMMF) Scholarly Activity Collaborative. The aim of FMMF is to build a strong family medicine workforce to provide high-quality, comprehensive care for the people of our region. We recruited family medicine residency program directors and faculty from the existing FMMF database of program directors in this region, which aligns geographically with the AAMC North Central area. ⁴ We emailed 117 of 118 program directors with available emails up to three times between January and March 2024.

A total of 24 respondents indicated interest in participation in a qualitative interview and completed a screening questionnaire with residency program and respondent demographic data via a Qualtrics survey. We added additional screening questions to inform the directionality of the qualitative interview guide (Appendix A). We then used a maximum variation purposeful sampling technique ¹³ to ensure that subjects represented characteristics of interest (state, rurality, community, program size, and academic status). A total of 21 respondents completed an interview (88%). Three respondents were not able to schedule an interview due to either time constraints or declination to participate.

We integrated key constructs from the RE-AIM framework ¹⁴ guided by answers to the screening questionnaire into a 15-item semistructured qualitative interview guide (Appendix B). Two team members (L.O., L.A.) conducted 21 interviews between January and April 2024. Following verbatim transcription, team members (L.O., L.A., L.H., J.P., S.W.) open-coded the initial interviews using Dedoose software. Two team members (L.H., L.O.) created a codebook informed by RE-AIM constructs. After 10 interviews, no additional codes were identified; additional interviews were conducted to achieve maximum diversity of participant characteristics. We analyzed the data using thematic analysis due to its flexibility in supporting inductive and deductive coding and its use of systematic engagement with the data to iteratively construct latent themes from the coded data. ¹⁵ The team convened regularly and used discussion to refine the final themes. ¹⁶

Our study team included family medicine educators with experience in academic and community settings and with primary degrees in both medicine and health professions education. Both interviewers (L.O., L.A.) have extensive experience in conducting qualitative research and interviewing techniques. We attempted, when possible, to assign an interviewer that had no existing relationship with the interviewee.

Of the 21 participants, the greatest percentage of respondents were from Illinois (7, 33%), identified their role as program director (15, 71%), and represented a community-based university-affiliated program (13, 62%). Many programs served medically underserved populations (16, 76%), and about half served urban (12, 49%) or suburban (11, 49%) areas. On average, respondents allocated 26% of interview spots to candidates who sent a preference signal, with a range from 5% to 60%. About half of respondents somewhat or strongly agreed that program signals (10, 48%) and geographic and setting preferences (11, 52%) add value to the current system. Few strongly or somewhat agreed that program signals (3, 14%) and geographic and setting preferences (1, 5%) will have a positive impact on equity, diversity and inclusion. Additional respondent characteristics are shown in Table 1.

We identified four themes with examples in Table 2: (1) Program faculty adopted the signaling programs strategically to complement their existing application review strategies; (2) Respondents perceived that program signals were implemented to reduce the volume of applications and lessen the associated burden; (3) Program signals did not impact diversity and equity, but geographic preferences may benefit community health; and (4) Modifications to the signaling program are recommended to optimize use in primary care specialties.

Theme 1. Program faculty adopted the signaling programs strategically to complement their existing application review strategies. Some respondents expressed doubt about the benefits of program signals and chose not to change their interview selection strategy for this first year of implementation, noting “we didn’t use them, like, as a cut off or anything like that” (P15). Other respondents described integrating program signals into holistic review. As one respondent noted, “I think our emphasis was really trying to think holistically, and these were just additional puzzle pieces that helped us figure out how to do that” (P02), explaining that the signal was weighted alongside other elements in a scoring rubric for holistic review. Other respondents described using program signals to prioritize applicants who might be a good fit with their program, noting “[It] has made it easier for us to figure out which applicants are genuinely interested in our program” (P01). Respondents used this data to adjust the priority of application review or interview offer: “I would describe how we went about using the signals as a way to slightly further inform our decisions of who to offer interviews to in the lower tier of applicants” (P20).

Some respondents used signals and preferences to allocate interviews to applicants in the middle or lower tiers of candidates after holistic review, as one respondent explained, “Because we had many more people than we could interview in that group is when we started prioritizing program signaling and geographic signaling over those who did neither” (P20). One respondent described this strategy as a successful way to invite applicants off the wait list: “I matched two residents who came off of my wait list for interviews because they signaled me, and they matched in our program after being ranked fairly high” (P19). One respondent described an unsuccessful strategy: “We were being a little bit generous with some of them to say, ‘Oh, you know they are not the best, but they signaled us, so let’s give him a chance . . . but I think we just have to be a little bit more objective in terms of like, what’s our interview criteria?” (P3), explaining that some candidates who sent a program signal were ultimately not ranked by the program.

Theme 2. Signals were perceived as reducing application volume and burden. As one respondent replied, “The problem folks were trying to solve was trying to identify . . . true interest . . . as opposed to being just part of the application bloat and the wide net that applicants are casting” (P19). Most respondents noted stable or decreasing application numbers, with one expressing relief about applicant quality: “We had perhaps a slightly decreased number of applicants this year, and despite that the quality of the top applicants was exactly the same” (P19). As a result, respondents were hopeful that signals would not only reduce application numbers but decrease the burden of application review, as one respondent described: “I think the interview process is so manpower intensive. It’s a huge number of hours during interview season. It’s always very stressful for our faculty. So, I think if this is developed more it could eventually be used to sort of reduce the number of hours that we’re spending interviewing and recruiting people who are really not interested in coming to us” (P15).

Theme 3. Program signals did not impact diversity and equity, but geographic preferences may benefit community health. Among applicants who sent a program signal, respondents described searching for those who had a clear match with the mission of the program: “And so there’s a lot of them you can’t read anything into. But sometimes you get those kind of cues that somebody has values that align” (P11). In contrast, respondents more frequently appreciated the use of geographic and setting preferences to offer interviews to applicants with local ties, especially when a free-text response explained the applicant’s reasoning in more detail. As one respondent noted, “When people have roots to the area, they just tend to excel better in residency because they have a support system, or they have family nearby, or their partners with them, and they can cope through, like, the harder situations” (P13). Finally, respondents were skeptical that program signals would address larger workforce concerns, adding, “The problem that we are trying to solve is diversity and getting underrepresented folks into our residency. . . . Signaling is not gonna solve that problem for us. But that’s the problem that at least I was trying to solve” (P05).

Theme 4. Modifications to the signals and preferences are recommended to optimize use in primary care specialties. Respondents largely favored a lower number of signals to identify genuine interest, noting, “You’ve only got five of them, and you use one on me. That means something to me” (P18). Some program directors noted that program signaling might increase the burden on primary care programs when excellent candidates apply to family medicine as a backup specialty, occupy a scarce resource in interview allocation, and then match in another specialty, requiring programs to use the Supplemental Offer and Acceptance Program (SOAP) process to fill their program: “I really worried about the fact that somehow we missed in their application the fact that we used a lot of effort trying to identify red flags and not waste that really valuable interview time resource on folks that really aren’t committed to family medicine” (P19). Suggestions for improvement included eliminating multispecialty signaling: “I would say you need to designate a primary specialty of interest, and you only get the signals for that specialty” (P08); similarly, about capping total signals, “One of the things that I would love to see is that there is a maximum number of signals, regardless of specialty” (P09).

Reflecting on their implementation strategies and experiences with signaling and preference programs, we asked program faculty for key recommendations to share with stakeholders (Table 3). Consensus among respondents revealed that most US medical school candidates will match into a top choice. Some respondents suggested that advisors recommend to applicants that signals should be used for reach programs rather than those already familiar with the candidate due to their local ties: “Try to use it if you need to establish some foot in the door or somewhere out there” (P18). Respondents desired dissemination of best practices within the specialty, with a consensus on whether applicants should signal their home programs. Finally, respondents called upon the AAMC to adjust the signaling programs for all specialties. Respondents requested a free-text box for applicants to explain why they were signaling a program and for consideration of interventions to reduce negative impacts of multispecialty signaling on the primary care workforce.

This qualitative study of family medicine program faculty in the Midwest United States demonstrated largely positive experiences with both program signals and geographic and setting preferences during the first year of their implementation and identified four themes that explained their experience. Program faculty implemented signals and preferences strategically to complement their existing application review strategies. They perceived program signals as having promise for reducing application volume and burden. Perhaps unique to primary care, program faculty prioritize fit with program mission and community health, and they expressed skepticism that program signals would help with these critical primary care workforce challenges. However, they identified free-text comments regarding geographic and setting preferences as an underutilized tool that could also be applied to program signals and challenged AAMC to address negative unintended consequences of multispecialty signaling on primary care.

Our findings align with prior studies showing general program director satisfaction with program signaling among early adopter specialties such as otolaryngology and orthopedic surgery. ^{17, 18} Our respondents also had largely positive views of geographic and setting preferences, aligning with a previous study showing a greater likelihood of interview offer for applicants who signaled and also had a geographic connection. ¹⁹ Given the dearth of qualitative studies on this topic, our results extend what is known by describing specific strategies used to implement these programs. Some best practices identified by respondents included integration of signals and preferences into existing holistic review rubrics and use of targeted stratification for lower tier and wait-listed applicants for interview selection.

Respondents perceived program signals as reducing application volume and burden, but additional data may be needed to quantify the impact of program signaling and to address the need for additional strategies to combat application bloat in primary care specialties. While some specialties have embraced larger numbers or tiered signals as an alternative to application caps, such strategies may not be perceived as necessary for family medicine. ²⁰ Preliminary data on the first year of signal implementation in family medicine (2023–2024) demonstrated an average decrease in applicant numbers per program of 19% and in the average number of programs applied to by an applicant of 14% (55 programs to 47 programs). ²¹ Further innovation in artificial intelligence technology and tools to filter applications via the AAMC collaboration with Thalamus may assist program directors in further identifying candidates who best match their program mission. ²²

Our results identified concerns with multispecialty signaling and offered suggestions for improvement. While program signals intend to identify genuine applicant interest, interviewees noted that for them to discern whether applicants preferred family medicine was nearly impossible, and they expressed surprise after applicants matched into another specialty. To strengthen the primary care physician workforce, our results support further investigation into the impact of multispecialty signaling and model possible changes such as a total cap on signals regardless of specialty or limiting signals to a single specialty. This process will become more complex as obstetrics and gynecology replace their participation in AAMC Electronic Residency Application Service (ERAS) with an independent application platform, the Residency Centralized Application Service (ResidencyCAS). ²³

One strength of this study is that our qualitative data provides in-depth insights that are missing from the existing published evaluation data from AAMC. ²¹ We limited this study to the Midwest states (aligned with the AAMC geographic North Central area) to provide data to support the community of practice of residency programs in the family medicine Midwest organization with building a regional family medicine workforce. This limited the total number of potential participants. We did not ask reasons for nonparticipation among those who declined an interview. Future studies may consider broader inclusion criteria, expanding to family medicine residency programs in other geographical regions and exploring program director rationale for nonparticipation. We enhanced rigor in the study by purposefully sampling to obtain a large variety of different residency program demographics, but our results may not represent the views of all program faculty in different geographic areas. To rapidly disseminate findings, we conducted most interviews before the 2024 match, limiting our ability to correlate our data with match results. Further quantitative and qualitative evaluation of these programs and their impact on the specialty of family medicine after subsequent interview cycles will add insights about best practices in implementation and modification. Finally, while we limited our focus to faculty, the experiences and perspectives of student applicants, advisors, and program coordinators are crucial considerations for policy suggestions.

Program faculty strategically implemented preference signals into holistic review to reduce application review burden. Further modification of the program signal and geographic and setting preference programs should support the unique experiences of primary care residency programs and the need for primary care physician workforce development. Future research should focus more on the impact of refining the signaling programs and the impact on match outcomes and SOAP participation rate.

This work was presented at the Family Medicine Midwest Conference, Naperville, Illinois, September 29 to October 1, 2023; and the 2024 Society of Teachers of Family Medicine Spring Conference, Los Angeles, California, May 3–8, 2024.

Dr Oshman reports stock holdings from Procter and Gamble, Johnson and Johnson, Merck, Abbvie, Abbott and Dupont, outside the submitted work.