Peer and learner evaluations of faculty in graduate medical education are critical for performance improvement, promotion, and resource allocation.1 Although data are limited, studies show that feedback from residents may help faculty improve teaching.2,3 Narrative feedback, in particular, can provide important contextual information.4 Existing reviews of narrative feedback from residents to faculty show that the feedback is often relevant, but not specific. This limits the utility of feedback to improve teaching.5-7
Multiple feedback tools exist to evaluate clinical teaching. However the likelihood of capturing narrative feedback varies among these tools.1 Barriers to the use of feedback tools include time-consuming processes, issues with accessibility, reliance upon rating scales, and the retrospective nature of solicited feedback. Mechanisms to solicit feedback are more effective when they do not require significant changes to workflow.8 To address these barriers, a mobile application, the Faculty Feedback Facilitator (F3App), was developed to allow for real-time capture of narrative feedback for faculty in the medical education setting. In 2017 to 2018, the F3App was piloted across eight family medicine residency programs. Training and technical assistance were provided to programs on implementation and best practices for feedback. Participating programs reviewed the F3App positively.9
This study expands the initial pilot9 by qualitatively examining the narrative feedback from the same eight programs.
Setting and Data Collection
We included narrative feedback observations entered about faculty by residents and peers from July 1, 2019 to June 5, 2020 across the eight programs in the analysis. Program characteristics are shown in Table 1. Resident observations were anonymous by default per the Accreditation Council for Graduate Medical Education (ACGME), however learners were able to deselect this option.10 Faculty peer feedback was not anonymous in order to foster transparency and trust, important elements in a robust culture of feedback.11 Programs were offered training on the Situation-Behavior-Impact feedback model and tools were implemented at the discretion of the program.12
Using a deductive content analysis approach, we coded each observation as “positive,” “constructive,” and “actionable.”13 The categories were not mutually exclusive and are based on previous work.14 We also coded observations according to perceived environment: “clinic,” “inpatient,” or “general/unknown.” Table 2 includes coding definitions.
The four coders conducted an initial independent coding of a subset of observations and then reviewed the codes as a team to create a shared understanding of the coding scheme. Next, the data set was divided in half and two team members coded each half. Upon completion, the full coding team discussed coding discrepancies to achieve consensus.
The University of North Carolina Institutional Review Board approved this study (IRB # 20-0324).
Three hundred thirty unique observations were generated during the study. Table 3 shows the coding summary. Thirty-three percent (110/330) of the observations were made by peer faculty and 67% (220/330) were made by residents. Most observations were positive (98%) and actionable (88%). Observations were made in clinic, inpatient, and general educational settings. Sample entries for each observation category are listed in Table 2.
Table 4 compares the types and settings of observations by faculty and residents. There was little variability in the types of observations recorded by faculty vs residents.
Table 5 compares observations submitted anonymously vs nonanonymously by residents. Sixty-one percent (134/220) of the observations were submitted anonymously. All eleven (100%) constructive comments were submitted anonymously.
Based on literature review, this is the first qualitative analysis of feedback to medical faculty from residents and peers using a mobile application. The findings demonstrate that the mobile interface yielded primarily positive and actionable/specific narrative feedback. This type of feedback encourages and reinforces positive behaviors and is consistent with best practices for providing feedback.15-18 Prior studies show that narrative feedback is often not specific.5-6
The few constructive comments from learners were submitted anonymously. Learners want the option to give anonymous feedback.19-20 Learners did choose to deselect the anonymous default for some positive feedback.
This study has limitations. First, the design does not allow for a conclusion about why the majority of narrative observations were actionable/specific (ie, causality). Second, we only included residencies that participated in the pilot, some of which incorporated training on providing feedback. This may have positively skewed the number of actionable entries.
In conclusion, the use of a mobile application-based tool allows for collection of narrative feedback by learners and peers for faculty in a variety of settings. The feedback provides faculty with actionable and constructive suggestions for how to improve and/or continue their current teaching methods.
Conflicts of Interest: Cristen Page, a coinvestigator on this study, serves as chief executive officer of Mission3, the educational nonprofit organization that has licensed the tool from which the data from this study were acquired (F3App), from the University of North Carolina. If the technology or approach is successful at some point in the future, Dr Page and UNC Chapel Hill may receive financial benefits.
- Fluit CRMG, Bolhuis S, Grol R, Laan R, Wensing M. Assessing the quality of clinical teachers: a systematic review of content and quality of questionnaires for assessing clinical teachers. J Gen Intern Med. 2010;25(12):1337-1345. doi:10.1007/s11606-010-1458-y
- van der Leeuw RM, Slootweg IA, Heineman MJ, Lombarts KMJMH. Explaining how faculty members act upon residents’ feedback to improve their teaching performance. Med Educ. 2013;47(11):1089-1098. doi:10.1111/medu.12257
- Baker K. Clinical teaching improves with resident evaluation and feedback. Anesthesiology. 2010;113(3):693-703. doi:10.1097/ALN.0b013e3181eaacf4
- van der Leeuw RM, Overeem K, Arah OA, Heineman MJ, Lombarts KMJMH. Frequency and determinants of residents’ narrative feedback on the teaching performance of faculty: narratives in numbers. Acad Med. 2013;88(9):1324-1331. doi:10.1097/ACM.0b013e31829e3af4
- van der Leeuw RM, Schipper MP, Heineman MJ, Lombarts KMJMH. Residents’ narrative feedback on teaching performance of clinical teachers: analysis of the content and phrasing of suggestions for improvement. Postgrad Med J. 2016;92(1085):145-151. doi:10.1136/postgradmedj-2014-133214
- Myers KA, Zibrowski EM, Lingard L. A mixed-methods analysis of residents’ written comments regarding their clinical supervisors. Acad Med. 2011;86(10)(suppl):S21-S24. doi:10.1097/ACM.0b013e31822a6fd3
- Kassis K, Wallihan R, Hurtubise L, Goode S, Chase M, Mahan JD. Milestone-based tool for learner evaluation of faculty clinical teaching. MedEdPORTAL. 2017;13:10626. doi:10.15766/mep_2374-8265.10626
- Snell L, Tallett S, Haist S, Hays R, Norcini J, Prince K, et al. A review of the evaluation of clinical teaching : new perspectives and challenges. 2000;862–70. doi:10.1046/j.1365-2923.2000.00754.x
- Myerholtz L, Baker HM, Rollins L, Page C. Utilizing the F3App to capture learner feedback about faculty teaching. Fam Med. 2020;52(4):262-269. doi:10.22454/FamMed.2020.225169
- 2019 Common Program Requirements (Residency). Accreditation Council of Graduate Medical Education. Accessed April 5, 2022. https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/CPRResidency2019.pdf
- Page CP, Baker HM, Myerholtz L. Using a delphi technique to define a feedback culture in graduate medical education. Fam Med. 2021;53(6):433-442. doi:10.22454/FamMed.2021.600416
- Bommelje R. The Listening Circle: Using the SBI Model to enhance peer feedback. Int J List. 2012;26(2):67-70. doi:10.1080/10904018.2012.677667
- Kyngäs H, Kaakinen P. Deductive content analysis. In: Kyngäs H, Mikkonen K, Kääriäinen M, eds. The Application of Content Analysis in Nursing Science Research. Cham, Switzerland: Springer; 2020. doi:10.1007/978-3-030-30199-6_3
- Page C, Reid A, Brown MM, Baker HM, Coe C, Myerholtz L. Content analysis of family medicine resident peer observations. Fam Med. 2020;52(1):43-47. doi:10.22454/FamMed.2020.855292
- Voyer S, Cuncic C, Butler DL, MacNeil K, Watling C, Hatala R. Investigating conditions for meaningful feedback in the context of an evidence-based feedback programme. Med Educ. 2016;50(9):943-954. doi:10.1111/medu.13067
- Bing-You R, Varaklis K, Hayes V, Trowbridge R, Kemp H, McKelvy D. The feedback tango: an integrative review and analysis of the content of the teacher-learner feedback exchange. Acad Med. 2018;93(4):657-663. Accessed July 16, 2018. http://links.lww.com/ACADMED/A484 doi:10.1097/ACM.0000000000001927
- Ende J. Feedback in clinical medical education. JAMA. 1983;250(6):777-781. doi:10.1001/jama.1983.03340060055026
- Gigante J, Dell M, Sharkey A. Getting beyond “good job”: how to give effective feedback. Pediatrics. 2011;127(2):205-207. http://pediatrics.aappublications.org/cgi/doi/10.1542/peds.2010-3351 doi:10.1542/peds.2010-3351
- de la Cruz MSD, Kopec MT, Wimsatt LA. Resident perceptions of giving and receiving peer-to-peer feedback. J Grad Med Educ. 2015;7(2):208-213. doi:10.4300/JGME-D-14-00388.1
- Robins L, Smith S, Kost A, Combs H, Kritek PA, Klein EJ. Faculty perceptions of formative feedback from medical students. Teach Learn Med. 2020;32(2):168-175. doi:10.1080/10401334.2019.1657869