Using the Objective Structured Teaching Ecounter to Assess Resident Teaching Skills

Residents play a critical role in educating clerkship students, and residency programs must ensure residents are prepared to effectively teach. Training residents to become educators is of growing interest in graduate medical education,1 and the family medicine training requirements from Accreditation Council for Graduate Medical Education (ACGME) state: “Residents must demonstrate competence in: educating patients, families, students, residents, and other health professionals.”2 However, residents’ teaching skills are rarely assessed, and no standardized assessment tool has been applied to family medicine residents.

The objective structured teaching encounter (OSTE) is a tool used for providing formative assessment of teaching skills.3-5 Traditionally utilized in faculty development, its use to train residents and fellows has been previously described.6-9 This study represents a first examination of the application of the OSTE to family medicine residents. The purposes of this study were to gather the qualitative impressions of family medicine interns regarding how the OSTE influenced their thoughts or feelings about teaching, and to gather reliability evidence for this OSTE rating scale.

Forty-one family medicine residents participated in the OSTE. Ten interns participated in the focus groups. Prior to the OSTE, faculty raters, residents, and medical students serving as standardized learners underwent training. To provide a framework for teaching in the OSTE encounter, residents received instruction on the five microskills.10 Because time is a significant obstacle to clinical teaching,11,12 this approach was chosen due to its time-efficiency. Residents read a paper on the five microskills13 and a PowerPoint presentation that provided detailed explication of each of the microskills, augmented with video demonstrations. Five faculty raters met for 1 hour as a group to review a sample case video, practice completion of the assessment form, and compare/discuss ratings. Six standardized learners underwent 1 hour of training that consisted of background on the purpose of an OSTE, explanation of the five microskills, watching a sample case video, and rehearsal of the case script. Students received a $50 gift card for their role as standardized learners. All training sessions were facilitated by one of the authors of this study who has experience in assessment and faculty development. Training materials/case script are available online at the STFM Resource Library,10 or directly from the authors.

Each live OSTE encounter simulated a precepting experience where the standardized learner presented a patient case about hyperthyroidism14 to the resident. As the resident responded to the scripted queries from the standardized learner, the faculty rater observed the educational interaction between the resident and the standardized learner. After each encounter, faculty had 5 minutes to complete the assessment form and provide residents with feedback on their performance. Encounters were videotaped and each was reviewed by a second faculty rater to examine agreement between raters.

The rating form was developed from resources published by Elizabeth Morrison on MedEdPORTAL and a tool developed by the Shapiro Institute at Harvard Medical School.15 In addition to seven items designed to measure general teaching behaviors, we included five items targeting the five microskills.

We used descriptive statistics to capture resident performance on the instrument. We used Cronbach a to characterize the internal consistency reliability of the instrument and percent agreement; we used weighted k16 and intraclass correlation to quantify interrater reliability.

We conducted two separate, nonsequential focus groups to assess the interns’ perceptions of the OSTE (see STFM Resource Library for focus group questions). To ensure that residents felt comfortable providing honest feedback, a researcher unaffiliated with the residency program conducted semistructured, open-ended focus groups, taking field notes and audio recording the sessions. No residency faculty were present.

We provided deidentified digital transcriptions of the recordings from a third-party transcribing service to two investigators for analysis. We conducted a thematic analysis of the transcription and fields notes utilizing immersion-crystallization.17The investigators reviewed the key themes to verify consistency and validity of interpretations. The University of Buffalo Institutional Review Board deemed the study to be Not Human Research (STUDY00003216).

Residents generally demonstrated evidence of the five microskills and other behaviors related to creating a supportive learning environment while being less proficient at encouraging self-direction and input from the standardized medical students. Table 1 shows performance.

Analysis of the focus group data yielded six emergent themes. Themes and illustrative quotations are shown in Table 2.

Internal consistency reliability was good (α=.77 [.65, .86]), as was interrater reliability. Out of the 492 total ratings (41 residents: 12 items), raters exactly agreed 396 times (80%). Of the 96 disagreements, the ratings were within one scale point 83 times. Weighted k was .71 (.65, .77), and the intraclass correlation was .82 (.66, .90) indicating substantial agreement.

Analysis of the OSTE data revealed that family medicine residents were capable of demonstrating favorable teaching behaviors while benefitting from participation in the assessment process. The OSTE provided residents with an opportunity to put their teaching skills into action and receive feedback on them, something that is not feasible when mere instruction in teaching skills is offered without an applied assessment process. The formative feedback residents received helped them feel more validated, confident, and better equipped to give constructive feedback to others during future teaching encounters.

Psychometric analyses revealed good internal consistency and interrater reliability. Statistical measures of interrater agreement in this study are comparable to those found by Zakoff et al3 with pediatric residents, suggesting that different faculty raters are able to apply the rating scale in a consistent fashion—an important consideration for programs considering implementation of an OSTE.

That only interns were able to participate in the focus groups represents a limitation of the qualitative data and the conclusions that may be drawn from them. It is possible that the benefits to teaching confidence reported by the interns would not generalize to residents in later program years who have had more applied teaching experience. If this is the case, however, the observation that participation in the OSTE is beneficial for residents who are just beginning their clinical teaching remains an important finding.

Another limitation of this study is the lack of follow-up data. For example, although participants demonstrated teaching skills and reported a boost in teaching confidence in the short term, there is no evidence to suggest that these would be sustained indefinitely. Recent research by Zakoff et al3 suggests that OSTEs are an appropriate approach for documenting longitudinal change in resident teaching behaviors.

Despite these limitations, this study represents the first application of an OSTE to family medicine residents using a reliable assessment tool. It is a promising approach for programs interested in assessing their residents’ teaching skills in accordance with ACGME standards, and provides useful formative feedback to bolster residents’ comfort in teaching.