Assessing Student Competencies in Antibiotic Stewardship and Patient Counseling

Antibiotic misuse contributes to antibiotic resistance and is a global public health threat.1,2 Annually, 2 million people acquire antibiotic resistant infections in the United States, and 23,000 die.2 In 2012, the American Board of Internal Medicine (ABIM) Foundation, Consumer Reports, and nine medical specialty societies established the Choosing Wisely (CW) campaign to promote resource stewardship and encourage conversations with patients about unnecessary care.3-6 Through CW, various specialty societies recommend against the inappropriate prescribing of antibiotics for upper respiratory infections (URI).^4,6,7

Several professional medical societies promote antibiotic stewardship education for medical students.3 Medical students want more education on antibiotic stewardship.7,8 The literature on antibiotics stewardship training mainly reports on feedback to learners in postgraduate training programs.⁹ Few programs for medical students report behavior change outcomes.^9,10 Although several instruments assess medical students’ general communication skills, none assess students’ ability to communicate about stewardship of antibiotics.^11-13 To address this gap we have developed, and determined the psychometric properties of, a novel instrument to assess students’ antibiotic stewardship competency.

Third-year students at the Albert Einstein College of Medicine (Einstein, n=96) and the Warren Alpert Medical School of Brown University (Brown, n=82) participated in this study during their family medicine clerkship (2017-2018). Both institutional review boards deemed this study exempt.

At both institutions, antibiotic stewardship instruction is integrated into the preclinical Microbiology/Infectious Disease course. Students reviewed the CW website and its related videos12 during the family medicine clerkship. At the clerkship’s conclusion, student competency attainment is assessed via an observed structured clinical encounter (OSCE).

We completed a multistage, iterative development and implementation of a CW-OSCE with trained standardized patients (SPs) as evaluators.14 At both institutions, the CW-OSCE was one of five end-of-clerkship OSCE stations. Each student interviewed and counseled an SP requesting antibiotics for URI symptoms using the framework of CW concepts.13 The SPs and simulated cases were different at each school: viral pharyngitis (Brown) and viral rhinosinusitis (Einstein).

Using the CW framework,13 we designed eight items to measure students’ ability to engage patients meaningfully in appropriate prescribing of antibiotics.12 For example, these items included whether the student explored patient’s reason for requesting antibiotics, explained the side effects of antibiotics, and planned a follow up. Each antibiotic stewardship concept was measured by one to three items, and each item had a yes (2 points), partial (1 point), or no (0 point) rating scale.15,16 All items measured antibiotic stewardship concepts rather than general communication skills.17 Table 1 gives the main-item concepts for assessing antibiotic stewardship. SPs also assigned a single-item global rating of the student performance (range from 0-2, with higher ratings indicating better overall performance for the station).

SPs at both institutions received training in the use of the instrument. The instrument is currently in use, and is available from the authors upon request.

Preliminary analysis of the initial Einstein data showed minimal variability in the student scores (SP gave perfect scores to almost all students). This prompted SP retraining and exclusion of the first 6 months’ data from the final analyses. We used the R statistical software program and the Psych package (R Foundation for Statistical Computing, Vienna, Austria) to calculate a total instrument score for each student, as well as item difficulties, score distributions, scale structure, reliability (Cronbach α), and concurrent validity via item-total correlations.20 We analyzed results separately from each school to generate initial evidence for external validity. Taken together, these analyses were intended to determine the psychometric properties of the instrument.18,19,21-24

Analysis of the instrument scores of all students during the study period (Einstein: 96, Brown: 82) showed that item difficulties averaged 1.6 (range 1-1.9), which is ideal for discriminating student ability. Cronbach α was 0.64 (Einstein, 95% CI: 0.53 to 0.74) and 0.71 (Brown, 95% CI: 0.60-0.79), both within a commonly accepted range (ie, 0.60-0.85) for estimating reliability (Table 1).

Table 2 shows that simple correlations (r) between items were in an acceptable range (ie, 0.15-0.50), considering that correlations deflate when item score ranges are limited to discrete values of 0 to 2. Except for items Q4 and Q5, with smaller Drop correlations due to the ceiling effect, item correlations were similar in magnitude for the two locations (eg, Q3 had relatively high, and Q6 relatively low correlations at both schools).

Global ratings and instrument scores were positively correlated (r=0.52, F [3, 174]=30.71, P<.001): higher instrument scores were correlated with more positive global rating scores. Table 3 shows that regression analysis predicting global scores based on total score and test site showed no significant interaction of scores by site, suggesting that the relationship between total score and global score did not differ by institution and only total scores predicted global scores (coefficient estimate=0.13, t[174]=9.24, P<.001).

Literature reports that objectively designed instruments can assess students’ contextual knowledge and communication skills in identifying issues impacting clinical outcomes.11 We have developed a novel instrument measuring students’ demonstration of antibiotic stewardship concepts.12 The reliability and item statistics results demonstrate that the instrument measured scores reliably and support its use for similar purposes at other institutions. Higher instrument scores predict higher global ratings, providing evidence of concurrent validity.

Our study has limitations. Students were assessed on a single case during their third-year family medicine clerkship, limiting generalizability to other educational settings and stages of training. Ideally, there should be coordinated preclerkship and clerkship curricula and assessments reinforcing antibiotic stewardship. Additionally, we cannot provide evidence of behavior beyond the clerkship nor in the clinical setting. Assessing these competencies in practice requires novel interventions such as unannounced SPs in patient care settings.25

We developed and tested a novel instrument to measure medical student antibiotic stewardship competencies. Implementation at other institutions with different cases and learners would provide additional validity evidence for the instrument and elucidate item performance under various conditions.