Content Analysis of Family Medicine Resident Peer Observations

Direct observation is a critical part of assessing learners’ achievement of the Accreditation Council for Graduate Medical Education (ACGME) Milestones and subcompetencies.1,2 It is well established that direct observations can provide timely, specific, and actionable feedback to allow residents to continue to develop.3,4 Despite this, there remains a paucity of direct observations to inform resident evaluation.5 Resident peer observations may have the ability to fill this gap, and align with the ACGME program requirement to use multisource evaluations (ie, from faculty, peers, staff, and patients). Research demonstrates residents are willing providers of peer feedback6-10 and that peer feedback is a feasible and reliable way to evaluate residents.11-13 The purpose of this paper is to explore the content of residents’ peer observations as they relate to ACGME Milestone subcompetencies.

The University of North Carolina (UNC) at Chapel Hill’s Family Medicine Residency Program is an academic program with 11 residents per class and two fourth-year chief residents. The residency program uses the M3App, that allows faculty and residents to enter into their phone, laptop, or computer a narrative description of an observation and assign it one or more subcompetencies; results are distributed to residents, advisors, and the program’s Clincal Competency Committee (CCC). M3App was originally developed for family medicine Milestones but is now available to all specialties for a fee, which helps to cover nonprofit costs. Additional details of the M3App’s use and function are described elsewhere.14,15

To examine the content of peer observations entered into the M3App, we tabulated resident peer observations recorded between the implementation of the M3App in June 2014 and November 2017 by postgraduate year (PGY) of the resident observed (learner), PGY of the observing resident (observer), and ACGME subcompetency. Fourth-year chief residents and other fellows (all denoted PGY4) provided observations but were not observed in the M3App system.

The UNC Chapel Hill Institutional Review Board determined this study to be exempt (IRB #17-3108).

We tabulated the frequency of observations for each of the ACGME subcompetencies and calculated the proportion of observations categorized under each subcompetency. Additionally, we tabulated the proportion of observations made by members of each PGY class and about members of each PGY class.

Using a deductive content analysis approach, the research team coded each observation on three separate dimensions: “positive,” “constructive,” and “actionable.” These dimensions are based on common perspectives regarding categorizing feedback, and definitions are found in Table 3. Following best practices of coding qualitative data,16 three researchers coded separate sections of the observations, with a fourth researcher then coding a subset of all to assess intercoder reliability. Through group discussion the team achieved consensus regarding conflicting codes and refined code definitions; observations were then recoded using the refined definitions. We compared numbers of observations within each category across all postgraduate years, using the χ2 test for independence and estimated odds ratios and 95% confidence intervals for two-by-two comparisons.

Our data include 886 peer observations made by 54 residents (Table 1) during inpatient and outpatient clinical and academic encounters. The most frequently observed competencies were in patient care (56%), communication (47%), and professionalism (38%), followed by medical knowledge (28%) and systems-based practice (24%). Practice-based learning and improvement (16%) was observed least frequently. Table 2 shows the subcompetencies by frequency of observation, with Communication 3 (C3, n=164) being the most frequent and System-Based Practice 3 (SBP3, n=29) being the least frequent (see Table 2 for a description of each subcompetency).

Figure 1 shows patterns in subcompetencies observed by residents in learners of each postgraduate year. On average, there were more observations on Patient Care (PC) 1-3 for first-year residents and more observations on average for PC4-PC5 for third-year residents. Notably, each subcompetency accounted for a greater-than-average number of observations of at least one PGY class, though none did so for all three classes. Similar to Figure 1, Figure 2 summarizes the proportion of observations by each PGY class. PGY-4 observers, for example, made more than the average number of observations on nine subcompetencies. The PGY-4 residents were more likely on average to make observations about PC2, PC5, and Professionalism 2. None of the PGY classes made more than the average number of observations of SBP4.

Table 3 shows proportions of observations made by and about each PGY class coded as positive, constructive, and actionable. On average, 97% of the observations were positive, 85% were actionable, and 6% were constructive. There were no statistically significant differences in the numbers of observations in any category between PGY1s, PGY2s, and PGY3s. Comparing PGY1-3 observations with those made by PGY4s, we found PGY4s equally likely to make constructive observations (OR 1.1–95% CI 0.75, 1.69), one-third as likely to make positive observations (OR 0.34–95% CI 0.10, 0.90), and more than twice as likely to make actionable observations (OR 2.49–95% CI 1.35, 4.87).

These results indicate that when asked to review their peers, residents provide comments that are primarily positive, which is consistent with published literature6,7 and has been shown to encourage and reinforce positive behaviors.17 They also show that comments provided on peer behavior are largely actionable, which is consistent with best practices for feedback provision.18,19 Furthermore, constructive observations were often imbedded in positive comments; many constructive observations indicated that the feedback was being documented formally in writing following in-person review, reflecting best practices for feedback provision.1 Residents tend to provide more feedback on certain subcompetencies compared to others; this may provide evidence that programs should rely on peers to provide feedback on certain subcompetencies but not on others.8,10 While we did not poll residents on why some subcompetencies are more commonly commented on than others, we hypothesize that these are simpler to understand and perhaps more commonly observed by residents.

Direct observation during residency training allows evaluators to more accurately assess a resident’s progression through the program and ACGME Milestones. Though previously driven by faculty feedback, several studies have identified that peer review is an important part of a residency program’s evaluation system. Peers are able to provide additional contextual information for CCCs regarding resident performance, and residents are able to observe behaviors and actions that faculty members do not, providing unique perspectives into resident performance.8,20,21 However, it is important to consider the implications of requiring peer feedback and the impact that may have on the important relationship dynamics among residents.6,7,22

Our results have limitations. The data are from a single family medicine residency program, limiting generalizability to other programs or specialties. The majority of the observations (71%) were also recorded by PGY-4 residents. This may also impact generalizability given that many residency programs do not have PGY-4 residents. Arguably, PGY-4 residents are not true peers, considering their leadership responsibilities. However, PGY-4 residents have unique relationships and observation opportunities with respect to other PGY levels. Additionally, even if many programs do not have PGY-4 residents, chief residents with additional responsibilities are a common feature of many residencies. Thus, the expectation of increased feedback from residents in leadership roles may be generalizable across other programs. In addition, we analyzed the data without regard to the accuracy of the subcompetency assignment. We did not solicit feedback on why residents choose specific subcompetencies or how comfortable or uncomfortable they felt about giving constructive feedback.

Despite these limitations, the results provide evidence for the content of resident peer feedback. Peers can provide perspective on the behavior and skills of fellow residents. Additional research should investigate the impact of peer feedback on behavior change.