Implementing an Anonymous Closed-Loop Feedback Process and Associations With ACGME Annual Survey Scores in a Family Medicine Residency

The Accreditation Council for Graduate Medical Education (ACGME) requires that residency programs elicit resident feedback and concerns.1 The ACGME surveys all residency programs annually on a variety of variables, including residents’ satisfaction with their program’s feedback mechanisms. The ACGME provides programs with their aggregate data and benchmarked national averages. Residents’ responses to these surveys are critical, as poor scores can adversely impact accreditation.

Specific guidelines for conducting feedback are absent, leaving much autonomy to individual programs. Traditional models of feedback, such as written surveys and small-group meetings,2 can be limited in timeliness, scope, and anonymity—all aspects that are important for effective feedback.3 Several programs have investigated anonymous feedback utilizing confidential focus groups, electronic suggestion boxes, and online surveys.4-7 One study comparing anonymous and open (identifiable) evaluations of residency faculty within internal medicine showed statistically significant lower scores in anonymous feedback compared to open feedback,8 suggesting that feedback format can influence results. Conversely, significant drawbacks to anonymous feedback include unfocused and reactionary comments that may not be representative of overall resident sentiments or constructive for meaningful change.7,9

In this study, we examined longitudinal changes in ACGME survey scores, before and after implementing an anonymous, closed-loop quarterly survey process. We hypothesized that an anonymous system allowing residents to regularly provide program concerns and suggestions, and discuss these openly in a group setting with administration, would correlate with higher ACGME survey scores in satisfaction and confidence with the program’s feedback mechanism.

The University of Utah Family Medicine Residency Program is a 3-year program with eight residents per class, which expanded to 10 residents per class beginning in 2018. We took data from the 2011-2019 annual ACGME resident survey aggregate reports. Survey responses from 2011-2012 served as baseline data. The intervention of the feedback process began in academic year 2012-2013.

Every quarter, (1) a web link to an anonymous online survey was emailed to all residents, providing opportunity to list concerns, suggestions, or other residency-related feedback; (2) responses were reviewed by the program director, core residency faculty, and the chief residents during regularly-scheduled meetings; and (3) survey results, as well as planned program responses, were summarized and discussed at a regularly scheduled resident business meeting. In this way, each feedback cycle was followed by closed-loop communication to acknowledge and address the survey comments. This quarterly feedback system was a supplement to already-established feedback mechanisms, including postrotation surveys, evaluations of faculty, annual program evaluation, and didactic evaluations.

This study evaluated four questions from the ACGME resident survey. The ACGME questions used a 5-item Likert scale (1=strongly disagree, 5=strongly agree), and were reported as an aggregate program score for each question. The ACGME survey reports aggregate national scores for program comparison. The statements evaluated are presented in Table 1.

We calculated relative differences for each averaged response between 2011-2012 and each ensuing academic year, reported as percentage change from baseline. We used simple linear regressions to show trends. We also plotted the relative differences against academic year to show the magnitude of change compared to the first-year score. We used one-sample z test to compare each year’s percentage change from baseline. We used two-sample t test to compare program average scores versus the national average, and we set significance at 5%. We performed analyses in STATA software, version 16.0 (STATA Corp, College Station, TX).

The University of Utah Institutional Review Board deemed this project nonhuman subjects research (#00130505).

Fifty-six percent of the 66 residents involved in the feedback process, across the 8 years were female, and predominately identified as non-Hispanic/White (90.1%).

The plots of average scores across academic years for the four questions are shown in Figure 1, and with the exception of Question 1 (Evaluations of Program Confidential), all show a stable or increasing trend across academic years.

Figure 2 shows the relative differences of each year’s average score compared to baseline scores from 2011-2012. Scores increased for all questions, except for one instance. In no cases did scores decrease from baseline. For the statement, “Satisfied that evaluations of program are confidential,” years 2013-2014, 2014-2015, and 2018-2019 showed significantly higher scores from baseline (2012-2013: 13.2% increase, P=.051; 2013-2014: 18.4% increase, P=.001; 2014-2015: 15.8% increase, P=.008; 2018-2019: 18.4% increase, P=.002). For “Satisfied that program uses evaluations to improve,” only year 2018-2019 had a significantly higher percentage change (16.2% increase, P=.008). For “Satisfied with process to deal with problems and concerns,” year 2012-2013 showed marginally higher change than 5%, and year 2018-2019 showed significantly higher change (2012-2013: 13.2% increase, P=.051; 2018-2019: 13.2% increase, P=.04). The last statement, “Residents can raise concerns without fear,” saw significantly higher percent changes in year 2013-2014 and year 2018-2019 (2013-2014: 17.9% increase, P=.002; 2018-2019: 25.6% increase, P<.001).

Table 1 shows the comparison of program average scores to the national scores across each academic year for the four questions. For the question “Satisfied that evaluations of program are confidential,” the University of Utah scored significantly lower than the nation in years 2011-2012 (baseline) and 2015-2016 (2011-2012: 3.8 vs 4.3, P=.01; 2015-2016: 3.8 vs 4.3, P=.03), but was on par in other years. No significant differences were seen in question “Satisfied that program uses evaluations to improve.” The program scored marginally lower in the baseline year 2011-2012 on “Satisfied with process to deal with problems and concerns” (3.8 vs 4.2, P=.05), and on par in subsequent years. Finally, the program scored marginally higher for “Residents can raise concerns without fear” in 2013-2014 and significantly higher in 2018-2019 (2013-2014: 4.6 vs 4.2, P=.05; 2018-2019: 4.9 vs 4.3, P=.005).

Improvements were seen in the ACGME annual survey results following implementation of an anonymous, closed-loop feedback process. Residents perceived this process as confidential and safe. Prior research has suggested that psychological safety is an important variable in resident satisfaction and learning,10 and anonymous feedback processes may be a useful contributor to this perception.

We observed small changes in residents’ belief that the program uses evaluations to improve, and in satisfaction with the process to deal with problems and concerns. Qualitative analysis to explore this phenomenon could prove beneficial, but was beyond the scope of this project. The small improvement may relate to residents learning more about program changes that occur as a result of the closed-loop feedback, and may also reflect frustration with aspects of the program that cannot be easily changed. Prior research examining the ACGME survey suggests that perceived responsiveness to resident feedback is particularly salient to many residents.11

Limitations of this project include the single residency program design, limiting generalizability. This study spanned 8 years of data, introducing many confounders in the work and learning environments (eg, duty hour changes, off-service rotations, shifting some elective rotations from intern year to a later year). Strengths of the project include its longitudinal nature and comparisons between the program’s own data and national data. Further investigations are needed within multiple residency programs, and should incorporate a qualitative component on the feedback process.