Using Standardized Patient Assessments to Evaluate a Health Literacy Curriculum

Increasingly complex health systems necessitate patients learn and routinely perform self-care skills. Recognizing and improving patients’ health literacy has emerged as a central health concern. The Institute of Medicine defines health literacy as “the degree to which individuals have the capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions.”1 Ninety million adults have limited health literacy and 75% do not disclose this to providers.2 These patients are likely to face worse outcomes, use services less effectively, and among elderly persons and those in rural areas, experience poorer health and higher mortality.3,4

Standardized patient assessments (SPAs) provide for direct observation, training, and evaluation of student-patient interactions during histories, physical examinations, and wrap-up. SPAs offer objective data on learner performance.5,6 The SPA performance evaluation is enhanced when paired with evaluations by trained raters.7 Student performance in a 32-hour health literacy curriculum using interactive seminars, patient-centered communication, and health literacy skills was assessed with SPAs.

Participants included 435 third-year medical students completing a required 4-week clerkship in family medicine at the Wake Forest School of Medicine (WFSM) for the graduating classes of 2013 to 2016. Participants signed informed consents approved by the WFSM Institutional Review Board. In week 1, students attend five case-based seminars focused on recognizing problems and solutions associated with low health literacy (Table 1).8-23 The curriculum includes two team-based teaching clinics (TBTCs). Paired students participate in the care of faculty members’ continuity patients with chronic diseases in 4-hour TBTCs. Students are evaluated during two 20-minute SPAs that include a focused interview, exam, treatment plan, and completing a “subjective, objective, assessment, and plan” (SOAP) note.

Eight faculty received training in the Common Ground model and health literacy skills curriculum. The Common Ground instrument addresses six criteria-based, patient-centered skills. Interview skills are measured by a global rating that incorporates comprehensive criteria for SPA (Table 2). In the original validation, the interrater reliability was 0.85 for the global rating and 0.92 for the overall checklist assessment.8

To evaluate health literacy competency, we used seven questions from the US Health Resources and Services Administration (HRSA) health literacy website (Table 3).24 Reliably trained raters scored students’ performance for SPA via digital recordings. Three times annually, raters received iterative feedback to assure that scoring was consistent with benchmarking.

After video recording, SPA data was deidentified. Reliably trained raters evaluated both performances using the Common Ground instrument8 and completed the health literacy skills checklist (Table 3). Paired Common Ground scores and health literacy checklists were evaluated for differences between SPA-1 and SPA-2 using SPSS.25 Repeated measures analyses of variance were performed separately for the Common Ground and health literacy measures. SPA-1 vs SPA-2 was a within-subject variable and student class was a between-subject variable. Pearson’s correlation was used to correlate Common Ground global scores and health literacy scores.

Participants were medical students in the classes of 2013 (n= 109), 2014 (n=120), 2015 (n=107), and 2016 (n= 98). Students had an average age of 26.8 years (range 24 to 43), and 54% of students were men. Overall, 72% of students were white, 11% African American, 8% Asian, and 9% other. For the classes of 2013, 2015, and 2016, 314 of 338 (93%) videos had both SPA sessions (data not captured for class of 2014 SPA-2).

A health literacy curriculum with SPAs using trained raters near the beginning and at the end of the clerkship was implemented. After SPA-1, and receiving 8 hours of training with health literacy and communication components, students’ global scores were in the low competent range on Common Ground and on the health literacy skills checklist. Common Ground scores analyses revealed a main effect of class, a main effect of first and second SPA sessions, and an interaction between class and session (Table 4). The classes of 2013 and 2015 showed significant improvements from the first to the second SPA session (Table 4). The class of 2016 improved, but the difference was not significant.

Health literacy scores showed significant main effects of class and SPA-1 versus SPA-2 (Table 5). Post hoc tests showed all classes had improved scores for health literacy skills from SPA-1 to SPA-2, attributed to communication skills. Post hoc examination of the main effect of class (SPA scores averaged across both sessions) revealed that the class of 2013 had higher health literacy scores than the other classes combined across both SPAs. The class of 2015 had higher average scores than the class of 2016. The Common Ground instrument scores were positively correlated with health literacy scores at each SPA session (Pearson correlation .45 SPA-1, .50 SPA-2).

Training medical students to assess health literacy is essential for effective patient communication. The curriculum offers a range of activities focused on the attitudes, knowledge, and skills for a learner to effectively create care plans with patients of diverse backgrounds and health literacy levels. The unbiased raters’ assessments using a validated scale increases confidence in the results. Our pool of raters remained stable over time, maintained their enthusiasm, and demonstrated sustained ownership regarding the success of this curriculum. Between SPA sessions, students participated in two TBTCs. Thereafter, both communication and health literacy scores increased from SPA-1 to SPA-2 for the classes of 2013, 2015, and 2016.

One limitation of this research is that the clerkship directors preferred an introduction to curriculum before SPA-1 for students to learn health literacy knowledge and skills before being evaluated. Health literacy and communication seminars are provided in the first 2 days of the clerkship. While we do not have a true baseline, scores between the two SPA sessions improved significantly.

This curriculum provided a robust medical student experience designed to improve care for patients with limited health literacy. Rater training provides a structure that other medical schools could adopt to enhance their evaluation of students’ skills. Other programs could use the curriculum described with only the cost of training raters and standardized patients. Since patients are expected to perform increasing self-care, learners need to know what patients understand in order to learn how to communicate effectively.