Using a Delphi Technique to Define a Feedback Culture in Graduate Medical Education

Feedback is an important part of the learning process, and has been shown to affect the clinical performance of physicians.1 Research focus has shifted from the needs and behaviors of feedback providers to focus more on needs and perceptions of feedback receivers. Multiple studies explore resident perceptions and barriers to cultivating a positive culture of feedback.2,3 Much of the literature discusses feedback culture of programs as enabling or inhibiting feedback, and some literature provides the broad recommendation to “improve the overall feedback culture.”4 Some have attempted to collect or provide definitions of feedback culture. For example, through focus groups and interviews, Ramani et al found that educators and learners define feedback as “one that communicated clear institutional expectations promoting regular two-way feedback conversations.”5 London et al describe a strong feedback culture as “one where individuals continuously receive, solicit, and use formal and informal feedback to improve their job performance.”6 J.C. Archer describes a feedback culture as one in which “feedback is embedded implicitly and explicitly in all activities and in which students feedback to teachers as well as teachers to students.”7

Similarly, literature provides recommended ways to facilitate a feedback culture within an organization (some of which are specific to graduate medical education [GME]). Ramani et al developed 12 tips to promote a feedback culture within medical education, focusing their tips on sociocultural elements that include the relationships between giver and receiver, as well as institutional context.8 Kraut and colleagues suggest six strategies for facilitating an effective feedback culture in medical education, ranging from specific operationalized suggestions, for example “create daily triggers for feedback” to broader programmatic goals, such as “differentiate feedback from assessment.”3 London et al provide guidance on organizational practices and interventions with three overarching themes related to feedback: (1) enhancing the quality, (2) emphasizing its importance within the organization, and (3) providing support for using it.6 Yet, with each of these publications and suggestions for facilitating a feedback culture, we could not find consensus in the literature on either what defines a feedback culture or what steps a program can take to facilitate such a culture.

Using case study methodology similar to Bodenheimer et al,9 we gleaned learnings from over 50 graduate medical education programs across the United States, most of which are family medicine residencies, to implement and sustain direct observation feedback tools.10–12 Through sustained work with 27 residency teaching practices in a regional primary care residency collaborative active in three states, Donahue, Newton, Page and colleagues collected data showing wide variation in clinical quality and utilization.13,14 Through this combined work, Page et al identified variation in program needs and general readiness to adopt feedback practices, and also discerned a possible association between a culture of continuous quality improvement and programs with enhanced elements of feedback practices. There are tools designed to assess cultural elements of continuous quality improvement in academic clinical practice, such as the Patient Centered Medical Home Assessment15 used in the Building Blocks of High Performing Primary Care,9 as well as tools that help to measure hospital organizational culture as it relates to improving the quality of cardiovascular care.16 Yet, there is no such assessment tool to assess a culture of feedback in family medicine residency programs or in GME in general. Given family medicine residency training includes multiple clinical training environments and microcultures in many specialties, we must start with the need to define a culture of feedback in the broader context of GME.

The purpose of this study was to establish expert agreement on the key required elements that define a robust feedback culture in GME. By using the Delphi technique, we harnessed the expertise of individuals in the field who have contributed to the considerable body of literature related to the topic of feedback and feedback culture.

To identify key elements of a feedback culture in GME programs, we used the modified Delphi technique as described by Humphrey-Murto et al.17 The Delphi method uses a panel of experts to anonymously come to consensus on a topic while minimizing the influence of group thinking processes.

We searched the literature for evidence around defining a feedback culture, focusing on literature that relates to graduate medical education. This review of the literature allowed us to identify previously defined components of a feedback culture from 10 different peer-reviewed journal articles, and organize them into various dimensions. We then reviewed each dimension and developed one single definition that accurately captured each. Throughout this process, we attempted to maintain original wording from the published literature as much as possible. Upon reviewing the definitions, it became clear that they naturally separated into two categories: (1) elements of a feedback culture, and (2) ways to operationalize a feedback culture. With these two categories in mind, we developed the first category comprising 15 elements, and the second category comprising 14 elements (Table 1).

We identified potential experts for our panel by reviewing the literature for existing, published definitions of a culture of feedback. We did not include published year as a limitation in our search, but the publications we found dated back as early as 2002. Though our search was not exhaustive, we felt that we had compiled a comprehensive list of published definitions from approximately 10 publications. From those publications, we invited the first, second, and senior authors to participate in the study as an expert panelist. We also asked these experts to nominate other possible experts. We invited potential panelists to participate in the study via email in May 2019. We did not offer incentives to panel experts. Our initial expert panel consisted of 30 individuals, all of whom had published scholarly articles on providing feedback and/or on feedback culture.

To minimize burden on our experts, we made an a priori decision to use a two-round Delphi process. The list of elements gleaned from the literature review was sent to our panel of experts using Qualtrics software (Qualtrics, Provo, UT), an online survey platform. We asked panelists to rate each element with the following ratings: “essential to a feedback culture,” “compatible with a feedback culture,” or “not important to a feedback culture.” We instructed them to consider their responses in the context of a graduate medical education setting. We also invited the experts to provide open-ended comments regarding each element. Panelist responses were confidential.

Using guidelines from Humphrey-Murto et al, where 80% is the highest minimum threshold suggested for use in Delphi studies,17 we set a minimum threshold of 80% agreement; elements that had 80% or higher agreement on the rating “essential to a feedback culture” were retained. Elements that had 80% or higher agreement on the rating “not important to a feedback culture” were discarded.

The research team consisted of two family medicine faculty and a research associate, all of whom have experience in the Delphi technique, and have previously been involved in, and published on, research related to giving and receiving feedback in graduate medical education.10–12 As a team, the researchers reviewed comments provided during the first round and made modifications to the wording of the items based on the panelists’ comments while working to maintain the original spirit of the element found in the literature. Table 1 shows the wording modifications. Based on feedback from the panelists, we also added two new items for round two: “Learners receive training on how to respond to both positive and constructive feedback which includes learning to recognize and manage emotions tied to receiving feedback,” and “Feedback seeking behavior is explicitly encouraged during orientation to a new organization/rotation.”

For the second round, each expert received an individualized survey, providing them with the aggregate panel rating, their own first-round rating, a collated list of open-ended comments provided during the first round, and description of any item revision. Experts used that information to rerate items using the same 3-point scale.  

This study was submitted to the University of North Carolina at Chapel Hill Institutional Review Board and received a determination of not human subjects research (IRB #18-2087).

For round one, we received 17 responses from panelist for a response rate of 68%. Three individuals declined, and we could not locate the correct email addresses for two individuals, which left 25 possible respondents from our initial pool of 30 identified experts. For round two, 15 of the 17 experts responded, for a response rate of 88%. Table 2 shows general demographics of our final expert panel. The majority (n=13) worked primarily in a university setting, and about half (n=8) reported working in GME. Two (n=2) panelists chose not to complete the demographic survey, therefore, their demographic results are missing. The panelists were diverse in their country of origin and highest academic degree.

Table 1 shows the ratings of items by the panelists for both rounds of the study. Agreement was reached for seven items in round one and an additional 10 items in round two. Thus, through the two rounds we obtained agreement on “essential to a feedback culture” on 17 elements. Table 3 shows the list of elements “essential to a feedback culture” per expert agreement. It should be noted that upon review of the final items, two items, “Feedback is provided in a trusting relationship” and “Feedback is embedded in a trusting relationship,” are very similar. We believe that these reflect the same construct and can be combined into one item.

We engaged a panel of experts utilizing an online Delphi technique to identify elements inherent to a strong feedback culture. Our process assisted us in pinpointing 17 behaviors and practices fundamental to a robust culture of feedback in a graduate medical education program. Our panelists identified universality, a supportive and trusting culture, emphasis on growth, and faculty role models as being essential elements of a feedback culture. In terms of operationalizing a feedback culture, our experts endorsed use of direct observation, time for review and reflection, rewards for commitment to a culture of feedback, encouragement of informal feedback, reflection and informed self-assessment, encouragement of feedback seeking behavior, instruction of learners and faculty on how to provide useful feedback, promotion of bidirectional feedback, and instruction for learners on responding to feedback.

Through open-ended comments, our experts also suggested that feedback must be coconstructed within the context of a meaningful and mutually respectful relationship, with the purpose being to challenge each other’s thought process or actions to support growth; this idea is also grounded in the literature.18 In other words, feedback is not necessarily something that one person holds to be self-evident and then disseminates to the recipient, rather it is a joint process of developing and then acting upon what was discussed. This feedback loop, or the act of taking feedback and changing behavior, is in turn an essential part of the process and must be taken into consideration when talking about developing a feedback culture. A consideration specific to the feedback process in GME brought up by our panel experts is that in medicine, sometimes patient safety supersedes the need for coconstructed feedback and people may need to change their behavior for the sake of patient safety, regardless of whether or not they agree with the feedback. Finally, our panel identified that role modeling is an essential part of developing a culture; in the field of GME, faculty are role models. Therefore, faculty must be actively engaged in the feedback process, rather than just checking it off the list. Programs can also work toward normalizing feedback conversations in order to develop a feedback culture within their programs; they must also work toward embedding feedback implicitly and explicitly in all activities, and both learners and teachers should be encouraged to actively seek out opportunities for feedback.7

Our findings can function as a guide for family medicine educators seeking to facilitate a culture of feedback within their programs. As identified in a review of the feedback literature by Bing-You et al, few articles describe how the culture of an organization can be assessed and improved.19 Using our essential elements and ways to operationalize a feedback culture, educators can evaluate their programs and institute changes that will encourage increased feedback between learners and faculty. By providing specific steps to operationalizing a feedback culture, we are presenting a framework for faculty seeking to facilitate a positive feedback culture in their programs.

Limitations of our study include the size of our panel of experts. A panel of 17 respondents is smaller than some peer studies, though Murphy et al suggest that a sample size of 12 is reasonable for a Delphi study.20 Similarly, our participants may not be representative of experts across the entire field of graduate medical education. We generated our initial list of experts from primary authors of papers on feedback in graduate medical education but did not seek to specify if this was focused on medical school, residency programs, or specific types of residency programs. We included experts from multiple countries (Table 2), and while we believe this enriched our review it is possible that different cultures have different perspectives on feedback; therefore results may not apply to all cultures globally. Finally, while we suspect many of our findings would be useful across the medical education continuum, some may not be generalizable to undergraduate medical education. Our intent was to have a broad view of feedback culture that could be applied across multiple specialties, given family medicine residency education occurs in multiple settings. Thus, the results may not be solely applicable to family medicine training programs. Also none of the experts identified as having a family medicine or primary care specialty. Nonetheless the authors’ experience in family medicine graduate medical education and faculty development leads us to believe it is indeed applicable to the discipline. Future studies could examine if there are differences in expert consensus related to graduate medical education specialty.

Experts theorize that a strong feedback culture fosters learning and growth in graduate medical education. Multiple studies focus on defining a culture of feedback, consider barriers to developing a culture of feedback, and discuss programs as encouraging or impeding a culture of feedback. Additionally, while tools exist to evaluate the culture of quality improvement in residency programs, no such tool exists to evaluate programmatic cultures of feedback. We address this in our study by utilizing a Delphi technique to reach agreement on essential elements of a feedback culture and ways to operationalize a feedback culture.

Our results enhance the current literature by defining an expert consensus on what constitutes a robust culture of feedback and providing a guide for educators on specific behaviors and practices for facilitating such a culture to foster programs conducive to learning and growth. Next steps include developing and validating an assessment tool using these identified essential elements. Using such an assessment tool will allow programs to evaluate their baseline and identify steps for improvement. This will also allow testing of hypotheses related to steps for improvement, associations between feedback culture and quality improvement culture, and impact on clinical outcomes. While evidence to support strategies for changing organizational culture is limited,21 using such an assessment tool will allow programs to understand where they are on the spectrum, and then also to take steps to improve the culture of feedback for faculty and learners alike.