Supplemental Offer and Acceptance Program Outcomes in a Family Medicine Residency Network

The National Resident Matching Program (NRMP) is the standard process for graduating US and international medical students to match with the residency program of their choice. While 95% of positions are filled in the Match, ¹ some applicants and positions remain unmatched. The Supplemental Offer and Acceptance Program (SOAP) is the NRMP process through which unfilled positions are offered by programs to unmatched applicants. ¹

Prior to 2012 the process to place applicants in unfilled positions was known as “the scramble.” ^1, 2 The experience was widely recognized as chaotic, involving multiple fax machines, applicants flooding clinics’ front-desk phones, and pressure to determine how to fill positions under a frantic time deadline.^2-4  Applicants and programs sometimes experienced commitments that were not kept, and an increasing number of applicants seeking the limited number positions created chaos; in 2010, the NRMP estimated that more than 13,000 applicants competed for 1,060 unfilled positions.²

In response, the NRMP introduced the SOAP program.² The SOAP brought order to the process, protected applicants and programs, and each year successfully places hundreds of unmatched applicants.^1, 5  The current SOAP process begins the Monday before Match Day, when schools, programs, and applicants are notified that they did not match. Over the next several days applicants look at programs with available positions, programs review applications from applicants, and programs and applicants speak with one another before a series of three to four rounds to match unfilled programs with applicants seeking a position.⁵  

There are few published data on the SOAP experience overall. Common findings in the literature are medical school experiences,⁶ national results,⁷ or specialty specific outcomes.^8, 9 The limited data about the outcomes and experience of programs that have matched residents through SOAP may in part be due to the confidential nature of the Match and SOAP processes. However, in our region, we have observed that programs are disappointed when they do not fill in the Match and participate in SOAP, and residents who are initially unmatched may be seen as less qualified or unlikely to fit into programs. As such, the purpose of this study was to describe residency program experiences with the SOAP process, including those of rural training tracks, and the residents who they matched with via SOAP.

We used a cross-sectional survey of program directors to examine the experience of participating in SOAP from the program perspective. The survey also aimed to measure program director perceptions of residents matched via SOAP in a large family medicine residency network over a 9-year period, from 2012 when the SOAP was initiated, to 2020. 

The survey was drafted and edited with input from one author, who had experience with the SOAP process as a former program director. It included quantitative and open-ended survey questions about the program’s primary clinical location and workforce mission and the experiences of the program personnel with the SOAP process. The main survey content included characteristics and comparators of residents who were matched via SOAP at this program as compared to residents matched through the main Match from their same cohort. In particular, we asked about resident graduation, need for remediation, leadership positions, culture fit, and whether the program director would hire or recommend them. We also asked them to rate the resident on Milestone performance in medical knowledge, professionalism, and interpersonal and communication skills.

Our network, the Family Medicine Residency Network (FMRN) comprises 31 family medicine residency programs and 10 rural training tracks across the five-state region of Washington, Wyoming, Alaska, Montana, and Idaho (WWAMI).  The number of residents recruited annually to the FMRN programs has steadily increased from 152 in 2012 to 249 in 2021. Annually, the FMRN tracks the programs that fill their resident slots on the first day of Match week and any that do not fill and move onto the SOAP process, which gave us background on the overall count of how many positions were in the SOAP across FMRN programs over the 9-year time period.  We identified and confirmed our data with NRMP archived reports to ensure accuracy of this count.

Using this information to inform our outreach, in the fall of 2020, we invited by email the 16 core program directors and seven rural training track program directors from FMRN programs who had ever participated in the SOAP to complete the survey described above. For the purpose of this study, we define rural training tracks (RTT) as training programs in rural settings with two to three residents per class and an affiliation with another larger program. In our outreach to the program directors to complete the survey, we attached the photo rosters of the program’s resident class(es) in the year(s) of SOAP participation to help remind them of who from that class had matched through SOAP as we did not know which specific residents were matched via SOAP. Program directors were contacted up to four times by email for their responses. We collected and managed study data using the REDCap electronic data capture tool hosted at the University of Washington.^10, 11 

We used descriptive analysis to compare demographics and other characteristics using Microsoft Excel and REDCap software. Due to the small number of study subjects and because the survey responses captured the entire cohort, statistical testing was not indicated. We also completed qualitative analysis of the two open-ended questions in the survey, which asked program directors about what they thought contributed to their needing to participate in the SOAP process and what advice they had for other programs. Using content analysis with an inductive approach, two of the authors (M.O. and A.W.) independently coded open-ended responses into common themes. These same two authors then met to compare their coding schema. There was strong overlap of themes by both authors, and differences were reconciled through mutual agreement. This study was approved as exempt from institutional review by the University of Washington Human Subjects Review Board.

During the study period of 2012 to 2020, 53 residents across the FMRN matched through SOAP out of a total of 1,658 residency positions offered. The number of residents matched via the SOAP in the FMRN ranged each year from one to six until 2019, when 12 residents matched through SOAP. Seven of the 23 programs that participated in SOAP were rural training tracks (RTT), and RTT resident positions (n=16) represented 30% of all residents matched via the SOAP despite the fact that RTT positions made up only 7% of all offered positions (111 of the 1,658). 

We received responses from all of the FMRN programs that participated in SOAP since 2012 (n=23). In these programs, there were 53 residents who matched through SOAP, and we received a complete response for 52 of them.

Programs that participated in the SOAP (ie, our survey respondents) varied in mission and structure. Nearly half of the responding programs’ primary clinical location was a federally-qualified health center (FQHC, 48%, 11/23). Seven were hospital-based programs and five were community-based programs. Respondents reported that most graduates go on to practice in the location/setting of the program’s workforce mission (mean 59%, median 66%), which for three-fourths of the programs is a rural training mission (n=17/23, 74%) and/or an underserved setting training mission (n=16/23, 70%, Table 1).

Overall, 87% (n=45/52) of the residents matched via the SOAP graduated or were anticipated to graduate from their residency program. This compares favorably to the FMRN as a whole, where approximately 95% of residents complete their training. Six residents matched via the SOAP who did not complete their initial residency program. Four of them left their program for another specialty, one was dismissed, and one left the program.

Five of these six residents who did not complete their initial residency program were among the 27 residents who matched via SOAP between 2012 to 2017 (and therefore should have graduated). Practice locations for 20 of the 22 residents who did graduate were known to programs; program directors felt that 77% of these residents were working in a setting that matched their program’s workforce mission (n=17/22).

Program director respondents reported a predominantly positive experience with their residents matched via the SOAP. Seventy-seven percent of the residents matched via the SOAP (n=40/52) were equally or less likely to need remediation compared to their peers, though nearly one-fourth were more likely to need remediation. Residents matched via the SOAP were perceived to be just as or more likely to take on a residency leadership position (77%, n=40/52). 

Respondents compared residents matched via the SOAP favorably to other residents in the specific Milestones of communication, professionalism, and medical knowledge at the beginning and end of their training. The majority were comparable if not better in these three domains at the beginning of their training. At the end of training (or timing of survey if training wasn’t complete), these ratings improved (Table 2).

Additionally, respondents were likely or very likely to hire or recommend 65% (n=34/52) of their residents matched via the SOAP after graduation. Twelve percent (n=6/52) of residents matched via the SOAP did not fit well with the culture of their program. 

Since RTTs participate in the SOAP at a disproportionate rate compared to core programs, we separately examined the RTT residents matched through SOAP. For RTT programs, residents matched via the SOAP also performed at or above par with their peers and we observed minimal differences between RTT residents matched via the SOAP and the overall sample (Table 2). 

The first SOAP experience is often challenging and unexpected for a program director. More than half of respondents reported feeling unprepared for the SOAP (n=13/23), though 35% (n=8/23) said they did feel prepared for their first SOAP. Program directors who participated in the SOAP in subsequent years all reported being better prepared.

When asked about advice they would give others in preparing for SOAP, respondents most often suggested to block off time for key program leaders to assist with SOAP. Other themes derived from their suggestions included the importance of understanding and preparing for the SOAP process, carefully screening applicants, and giving attention to the plan for communicating with applicants (Table 3). 

We also asked directors what factors they felt contributed to their program going through SOAP. Program directors at new residency programs cited lack of experience as a factor, though the most commonly cited issue among all respondents was poor preparation for the Match process itself. Examples included needing to refine the interview process or not interviewing enough applicants. Program issues such as leadership transitions or poor resident morale were also commonly reported factors. Finally, rural program directors reported challenges with finding applicants who were interested in rural training and especially the prospect of moving after the first year of residency, a common feature of the rural training track model.

We examined the experience of SOAP in a large network of family medicine residency programs over a 9-year period: a unique opportunity to gather some information about a complete group of residents who matched via the SOAP process. The majority of FMRN programs participated in SOAP at least once during the time period. Overall, their reported experience with residents matched through SOAP was reassuring. In general, residents matched via the SOAP were perceived as comparable or better prepared than other residents in their programs, were equally or more likely to take a residency leadership position, and were equally or less likely to need remediation. Most of these residents graduated, and most went on to practice in a setting that matched the workforce mission of the program. We also found that rural tracks were more likely to place residents via SOAP than nonrural programs.

The Match can be a daunting process for any program director, let alone for a new residency program. Program directors can prepare for SOAP by familiarizing themselves with the NRMP website and resources and setting aside time for themselves and key personnel during the process. Over one-third of program directors in our study reported feeling prepared for the SOAP the first time they participated and all directors who had to use the SOAP more than once were better prepared the second time around. The FMRN offers an educational session for new program directors every spring in advance of the SOAP and Match to help share wisdom and best practices.

This study is limited by our sample of family medicine residency programs in our geographic region. However, this allowed us to examine the unique SOAP experience of rural-focused programs and RTTs, given the rural nature of our region and its numerous rural programs. It is possible that SOAP experiences for other parts of the country may differ. While we only examined family medicine programs, we note that there are more than 700 family medicine programs that participate in the NRMP and many of them will experience SOAP at some point. Experiences of other specialties in SOAP may be very different. Our survey methodology has the potential for recall bias and subjective assessment; we have attempted to mitigate these issues by reporting data descriptively.

The programs that participated in the SOAP from our large network of family medicine residency programs over the last 9 years of the program demonstrate that the vast majority of residents matched via the SOAP are well prepared for training, contribute to their programs, and perform as well if not better than other residents. While it can be disappointing to go unfilled in the main Match, program directors can be reassured by these data. Additionally, while rural tracks are more likely to place residents via the SOAP than nonrural programs, both rural tracks and nonrural programs can prepare themselves for SOAP by familiarizing themselves with the NRMP website and resources and setting aside time for themselves and key personnel during Match and SOAP week in case they need to participate in the SOAP.