Clinical Educator and Student Perceptions of a Weekly Observed Practice Activity Evaluation Tool

Medical schools rely on clinician educators to teach.1,2 Clinician educators, henceforth referred to as preceptors, consist of university-paid and community-based clinicians. A tension exists for preceptors due to competing demands of patient care versus teaching, regardless of being paid by the university or as an unpaid, private clinician.3

An integral responsibility of clinician educators is completing evaluations. However, previous studies indicated preceptors were unclear of medical student expectations, making it challenging to complete evaluation tools and follow grading criteria.4,5 Students have also expressed concerns over the subjectivity of evaluations, which could impact their final grade.6

Therefore, we developed an instrument students use to meet objectives, receive feedback, and address accreditation comparability elements.7 Reviewing competency-based frameworks8 led us to identify a workplace-based evaluation we adapted, the Observed Practice Activities (OPA) checklist,9 to align with our required clinical skills.

Our goal with the OPA cards was to improve preceptor and student awareness of expectations during the Community Based Longitudinal Care (CBLC) course. This study reports perceptions about the OPA cards from preceptors and students.

The study took place at a public medical school in the Southeastern United States. CBLC is a required 16-week course, integrating outpatient family medicine, internal medicine and pediatrics clinical experiences. At the branch campus, CBLC integrated these disciplines over 24 weeks and included outpatient gynecology and psychiatry.

Using a consensus-building technique,10 a list of primary care clinical skills emerged from our objectives and published skills.8 The initial list compiled by course leaders included over 120 items. We consulted community-based preceptors to refine the list, resulting in a total of 113 items (see Appendix A, at: https://journals.stfm.org/media/4225/beck-dallaghan-appendix-a.pdf). We organized items by week, requiring students to complete a card each week for 15 weeks. We organized the OPA cards so skills increased in complexity, corresponding to didactic sessions (Figure 1).11 Preceptors had to observe and initial these skills, allowing students to receive feedback at the time of the activity.

We trained preceptors and students how to use the OPA cards12 in faculty development meetings and at course orientation. Full credit was given for completed cards, which contributed to their final grade.

Anonymous preceptor and student evaluations asked questions related to usability, direct observation of students, and understanding expectations. Scales ranged from 1 (lowest rating) to 5 (highest rating). See Table 1 for different scale anchors. We analyzed numeric responses using independent samples Mann-Whitney U tests with IBM SPSS v 25 (Armonk, NY).

Thematic analysis identified patterns within the narrative data.13 One investigator (G.L.B.D.) coded comments. The first author (K.B.F.) reviewed the coding scheme. We reviewed codes to identify common themes from preceptors and medical students.

This study was reviewed and approved as secondary data by our office of human subjects review.

All 149 medical students completed their OPA cards. Approximately 20 had problems completing the card, which was addressed at midcourse feedback sessions. Alternative experiences were scheduled to ensure completion. Seventy-nine of 115 preceptors (68.7%) completed evaluations; 80 of 149 students (53.7%) completed evaluations (Table 1).

Although students (60%) and preceptors (70%) indicated the OPA cards clarified expectations, preceptors found the cards of greater value (U=2,435.00, P=.008). The preceptors (88%) found the cards easy to use, as opposed to 58% of students (U=1,969.50, P=.001). Preceptors felt using OPA cards encouraged observation more than students (U=2,431.50, P=.004). In addition, 90% of the preceptors indicated OPA cards did not slow patient flow in clinic (Table 2).

Analysis of narrative comments (38 students, 22 preceptors) resulted in 40 unique codes coalescing into four themes: strengths of OPA, OPA items, OPA format, and perceptions of OPA. Table 3 lists themes and exemplar quotes.

For strengths of OPA cards, preceptors emphasized the benefits of being reminded about expectations. Students found the cards enhanced their experiences, specifically noting:

…they were also helpful for the specialty days as these days were more limited and it helped bring up conversations about the important topics in that field.

Although the OPA cards were helpful, the specificity and volume of the items presented challenges. The OPA card format was awkward, with suggestions to incorporate the OPA into a mobile platform.

Comments revealed divergent perceptions about OPA cards. Preceptors and students felt this was one more task to complete. Preceptors noted that when students were proactive, it reminded them to help students meet requirements.

In this study, we explored preceptor and student perceptions of a clinical skills observation card. Similar to other studies, preceptors did not feel the OPA cards burdened their clinical schedules.2,14,15 A key finding was that students and preceptors felt better informed about the expectations. Preceptors felt more confident teaching because they had a specific list from which to teach.4 Students appreciated knowing what was expected of them in clinic11 and the list was particularly helpful in clinical settings where preceptors rarely teach students.

Preceptors and students indicated items on the OPA cards were either not relevant or too specific. For this reason, a small number of students struggled completing a task at which time we scheduled alternative experiences.

The weekly format was felt to be cumbersome, particularly since there is no guarantee tasks on a weekly card will present in clinic that week. Students expressed concerns that items seemed repetitive; however, competency-based medical education is intended to improve the quantity and quality of feedback over time and in different contexts.16 Even though a consensus process was employed to generate the OPA cards, after implementation it became evident that regular review of the items is necessary, and adjustments need to be made. Future work with the OPA cards will include organizing items better and clarifying items perceived to lack relevance.

This study was performed at a single institution where expectations of students’ clinical skills may not be similar to other medical schools. However, identifying core skills using a consensus process could be replicated by other schools on longitudinal or block clerkships.

Based on our evaluation, preceptors found the OPA cards easy to use and did not slow them down during their clinical work. Students and preceptors found the cards to be helpful to understand expectations. There were logistical and item content issues noted that continue to be reviewed annually. Our OPA cards can be adapted by other schools by coming to consensus on core clinical skill requirements.