Limited Agreement Regarding Outpatient Preceptor Behaviors: Observer, Residents, Self

The greatest factors impacting a medical learner’s education are clinical teaching and role modeling, making effective methods for improving faculty teaching critical.^1,2 Validated assessment tools enhance clinical teaching by providing feedback to faculty on teaching skills.³ Faculty assessment tools are limited by their application to specific clinical settings,⁴ reliance on upward feedback,^5,6 use of perceptions rather than observations,^7-9 and lack of psychometric properties¹⁰ or clear theoretical framework.

To fill the gap of a tool tailored to outpatient family medicine training, we created and initially validated the Mayo Outpatient Precepting Evaluation Tool (MOPET).¹¹ A drawback of the MOPET is the time required for peer observation, impeding practical use.

Modifying the MOPET for self-reflection could decrease reliance on peer observation. A well-developed self-reflection framework is Donald Schön’s (1983) theory,¹² which includes retrospectively analyzing self and committing to future change.¹³ Self-reflection is common in health care to promote independent growth;¹³ therefore, it is a potentially helpful method for improving faculty teaching skills.

Additionally, resident-to-faculty feedback on clinical teaching is required by the Accreditation Council for Graduate Medical Education.¹⁴ A resident-based feedback tool could help meet requirements while reducing reliance on peer observation.

Our study aimed to examine how self- and resident-reporting of teaching behaviors corresponded to the original peer-observation MOPET.

This study was conducted at the Mayo Clinic Family Medicine Residency–Eau Claire, Wisconsin, program. The project was approved by Mayo Clinic’s Education Research Committee and deemed exempt by the Institutional Review Board. All residents (N = 15), core faculty, and community preceptors were invited to participate. Participation was voluntary, with no penalties or incentives. Resident-identifying information was collected only when residents were the observers.

The program’s behavioral scientist, a faculty volunteer, and a resident used the original validated version of the MOPET (MOPET-Observer) to observe a preceptor for a half-day clinic (session). All observers were given standardized instructions for how to use the MOPET. As described previously, observers tallied how frequently a teaching behavior occurred over the course of a session.¹¹ A behavior could be counted only once per teaching encounter, defined as an instance of a resident entering the precepting space to discuss the care of a patient(s).

At the end of the session, the preceptor completed a modified MOPET (MOPET-Preceptor), asking whether specific teaching behaviors were used at least once during that session. Each precepted resident completed a modified MOPET (MOPET-Resident) at the end of the session, asking which precepting behaviors occurred during that session. All MOPET versions included identical items presented in the same order.

Observer data was dichotomized as “absent” if not observed or as “present” if the behavior was observed at least once during the session. Resident notation of preceptor behavior was rated as “present” if at least one resident observed the behavior.

Interrater reliability between observer and preceptor, observer and resident, and preceptor and resident was assessed via the κ (kappa) statistic (range –one to+one), which accounts for chance agreement.¹⁵ A κ statistic can be calculated only if at least one observation is in each cell of the 2×2 contingency table. If variation is inadequate—for example, 100% prevalence—then a κ statistic cannot be calculated. Statistical analyses were performed via JMP version 18.0.1 (SAS Institute) with statistical significance set at P=0.05 without adjustment for multiple comparisons. Quantitative data lacked a normal distribution and is presented accordingly.

Across 48 sessions, 15 unique preceptors were observed two to four times each. The resident composition of sessions varied, with the number of first-, second- and third-year residents ranging from 0 to 3, with a median of 1 from each class. Sessions included five to 21 teaching encounters (IQR: 10–18; median: 15). The MOPET-Preceptor was completed for all but one session. Resident data were available for all sessions, coming from a single resident for two sessions, two residents for 16 sessions, three residents for 21 sessions, and four residents for nine sessions.

Table 1 shows how frequently teaching behaviors were observed or noted at least once during a session. Compared to the observers, residents more often reported experiencing 14 of the 20 behaviors at least once in a session by an average of 24%. Residents reported lower prevalence of only two behaviors relative to observers: didactic teaching and use of multimodal learning aids. Preceptors self-reported using 10 behaviors less often and eight behaviors more often compared to observers. Residents reported more often experiencing 18 behaviors compared to preceptors’ self-assessment.

Table 2 shows the agreement among observers, preceptors, and residents. We found statistically significant agreement between observers and preceptors for five of the 20 teaching behaviors (κ = 0.29 to 0.57); agreement between observers and preceptors could not be calculated for five behaviors due to 100% prevalence. Observers and residents significantly demonstrated agreement for 2 of the 20 behaviors measured (κ = 0.22 to 0.31), with nine behaviors lacking variation. Resident-preceptor agreement reached significance for three of the 20 behaviors (κ = 0.17 to 0.31), with agreement incalculable for eight behaviors.

This study evaluated how well self- and resident-reported teaching behaviors corresponded to the validated, peer-observation–based MOPET. Among preceptors and observers, we found moderate agreement for discussing how to support a resident with their learning goal, checking in on the learning goal, and asking about a differential diagnosis. We found weak agreement among preceptors and observers for handling interruptions well and the use of multimodal learning aids. No significant agreement existed for half of the 20 teaching behaviors. We did not find evidence that preceptors tend to over- or underestimate their overall use of various teaching behaviors. In conclusion, self-assessment by preceptors showed limited agreement with observers using the validated MOPET.

Residents noted teaching behaviors more frequently than observers did. Resident assessment weakly corresponded with observers for preceptors’ discussion of how to support the resident in their learning goal and use of multimodal learning aids. Residents’ overreporting of positive teaching behaviors could be due to our method of data pooling from multiple residents for a single session and/or favorability bias. Residents may have avoided negatively portraying preceptors, despite assurance of anonymity. Our findings provide further evidence that power differentials influence resident-based assessments, which makes their ratings less reliable.¹⁶

Our study found a high prevalence of multiple teaching behaviors such as demonstrating interest, being available to precept, and prioritizing precepting (98%–100%); the ceiling effect of preceptors consistently exhibiting these teaching behaviors limited our ability to calculate agreement. If other programs show that the same teaching behaviors are consistently prevalent, then an abbreviated MOPET excluding the most commonly observed behaviors may be more useful by reducing burden while maintaining validity.

This study is among the first to adapt a validated peer-observation tool for both self- and precepted resident reporting of teaching behaviors, allowing for direct comparison. Recall bias was limited due to data collection immediately following each session. This study examined occurrence of teaching behaviors rather than perceived teaching quality. Our single-site design limits generalizability.

Future studies replicating this work in other settings could further the MOPET’s validation and generalizability. A revised MOPET could focus on less consistently used teaching behaviors to reduce ceiling effects. Overall, while self- and resident-reporting tools offer practical advantages, direct peer observation remains important for reliable, accurate assessment of teaching behaviors.

This study was partially funded by the 2024 Endowment for Education Research Award from the Mayo Clinic College of Medicine and Science.

The authors thank all those who supported and participated in this study, including faculty, preceptors, and residents.