Creation and Initial Validation of the Mayo Outpatient Precepting Evaluation Tool

Effective teaching, requiring feedback and deliberate practice, is central to medical residencies. ¹ Skeff’s Stanford Faculty Development Program (SFDP) is a fundamental medical teaching framework ² that has demonstrable positive effects on teaching knowledge, skills, and attitudes, ³ as well as student and resident evaluations of clinical teaching. ^{4, 5} No preceptor assessment tool applies well to all clinical settings. ⁶ Many tools reflect learners’ perceptions rather than observation of precepting. ^7-9 Peer review is beneficial and less biased because it sidesteps the issues of power differential. ^{10, 11} Peer teaching observation tools have been validated in settings that do not generalize well to outpatient family medicine (ie, inpatient internal medicine residencies). ^12-14 The behavior-based precepting tools for outpatient family medicine were validated on medical student supervision ¹⁵ and used preceptor self-assessment,¹⁶ not peer assessment. We therefore sought to create and start validating an evidence-based, peer-observation instrument to assess precepting behaviors in a family medicine residency outpatient clinic.

We developed and initially validated the Mayo Outpatient Precepting Evaluation Tool (MOPET) in the outpatient clinic at the Mayo Clinic Family Medicine Residency–Eau Claire program in Wisconsin. This 5-5-5 program had six core physician faculty members (including the program director), one core behavioral scientist faculty member, and four community preceptors. Mayo Clinic’s Education Research Committee (#21-092) and Institutional Review Board (#21-009471) deemed the study exempt from approval.

While using the Patient Centered Observation Form–Clinician Version (PCOF) ¹⁷ to observe and provide feedback to residents in the continuity clinic, faculty preceptors inquired about a comparable tool for them. A literature review led to Beckman and colleagues’ work. ^{12, 14, 18, 19} Finding no tool directly applicable, we created the MOPET.

Beckman granted permission to use and modify the various versions of the Mayo Teaching Evaluation Form (MTEF). ^{12, 14, 18, 19} We selected and revised MTEF items that consistently demonstrated strong validity. We also created new items derived from the SFDP, ² PCOF, ¹⁷ Maastricht Clinical Teaching Questionnaire, ²⁰ and personal experience. Four core faculty members’ edits of this amalgam resulted in the studied version of the MOPET (Supplemental Figure 1). All items defined an observable, effective teaching behavior and were made quantitative to increase objectivity and reliability by counting the frequency of each behavior.

Each observation occurred over a half day of outpatient clinic, allowing for observation of multiple residents and presentations. The behavioral scientist and one volunteer faculty physician observed a precepting physician and independently counted the frequency of teaching behaviors. A precepting encounter was defined as a time a resident entered the precepting space to discuss the care of one or multiple patients. An observed behavior could be tallied only once per precepting encounter. At the end of the half day, observers used the MOPET to share feedback with the preceptor. The behavioral scientist also completed the SFDP-26 for each half day.

We estimated needing 20 observations to achieve at least 90% power to detect a correlation of 0.70 with a type-I error rate of 5%. We quantified the interrater reliability using Spearman’s rank correlation coefficient. We applied the Mann-Whitney U test to determine whether the behavioral scientist’s observations were significantly higher or lower than the physicians’. Using Spearman’s correlation, we assessed the cross-validity of the tool between items of the SFDP-26 and the sum of the relevant MOPET items (Supplemental Table 1). We identified relevant MOPET items (or lack thereof) by consensus. We chose nonparametric tests because only two of the MOPET items and none of the SFDP-26 items had evidence of a normal distribution (Shapiro-Wilk Test, P>.05). We performed all analyses in BlueSky Statistics (R version 3.6.3). ²¹

Five core faculty members applied the MOPET as an observer alongside the behavioral scientist in 20 precepting sessions of six different preceptors. The most observed behaviors were showing genuine interest, being available and ready to precept, and allowing the resident to present without interruption (Table 1 ). The rarest behaviors were encouraging the resident to ask questions, acknowledging when the resident taught them something, encouraging the resident to pursue the literature and/or other resources, and reinforcing self-directed learning.

The high interrater reliability of the MOPET items is presented in Figure 1. However, observers did not agree on when the preceptor was supporting the resident in activities the learner found difficult. We found weak concordance for prioritization of precepting over other tasks and explaining why the resident was correct or incorrect. The behavioral scientist observed more instances of preceptors supporting the resident in activities the learner found difficult and the use of multimodal learning aids, and fewer explanations for why the resident was correct or incorrect (all P<.05).

The good cross-validity between the SFDP-26 and relevant MOPET items is presented in Figure 2. We found especially strong correlations between the SFDP-26 and MOPET for items related to resident goal formation, use of multimodal learning aids, evaluation of a learner’s ability to analyze and synthesize knowledge, and delivery of feedback. We found no evidence that the MOPET measures correlated with any of the following SFDP-26 items: expressing respect for learners, discouraging external interruptions, explaining relationships in material, or promoting self-directed learning.

Our work began the validation of an objective precepting evaluation tool for outpatient family medicine residencies, filling an important gap in family medicine education. High interrater reliability indicates that the MOPET measures consistently recognizable behaviors. Strong correlations between the MOPET and most SFDP-26 items demonstrate cross-validity. Important to note is that the SFDP-26 is a peer’s gestalt of precepting, whereas the MOPET quantifies effective teaching behaviors. Because observers were not formally trained, the interrater reliability was likely lowered; yet our findings suggest that the MOPET requires little training.

Though this study was not powered to observe behavior change among individual preceptors, the behavioral scientist and observed preceptors expressed a strong sentiment of subjective improvement. Additionally, feedback from faculty members during MOPET development and unsolicited praise from observed preceptors regarding the tool’s value during the end-of-session feedback all signified face validity.

The lack of agreement between how frequently the behavioral scientist and peer observer saw the preceptor support a resident in activities the learner found difficult was surprising, but could be improved by including more concrete examples of this behavior in future versions. Another limitation of this study was the element of human error; for example, the correlation among observers for the number of residents in each precepting session was not 1.0 (possibly because additional residents not working in clinic came in to discuss a prescription request or patient case). Although measuring behavior frequency increases the reliability of measurement, it may not be an optimal approach for evaluating precepting behavior for constructive feedback; the best teaching involves using the right combination of behaviors for the specific learner at specific times. Furthermore, evidence-based precepting behaviors are tallied with this tool even when used at inopportune times, which could lead to a less effective educational experience. Lastly, which observable behaviors are most important to target is unclear.

Future research could include (a) re-evaluating after refinement via a modified Delphi method or factor analysis; (b) measuring use of the tool’s effect on preceptor behavior, learner experience, and patient outcomes; (c) assessing external validity in other family medicine residency programs, primary care clinics in other specialties, or other learners; and (d) comparing use to other preceptor improvement tools.

In conclusion, we created and initially validated an evidence-based, peer-observation instrument to support faculty development in the quintessential learning environment of family medicine residency: the continuity clinic.

Poster presentation, STFM Annual Conference, Indianapolis, IN, May 2022.

This study was partially funded by the 2022 Endowment for Education Research Award from the Mayo Clinic College of Medicine and Science.