Description and Evaluation of an Evidence-Based Residency Curriculum Using the Evidence-Based Medicine Environment Survey

Evidence-based medicine (EBM) education is required by the Accreditation Council for Graduate Medical Education.1 Practice-Based Learning and Improvement-1 encapsulates EBM competencies and is harmonized across specialties.1 Best practices for EBM curricular content and methods are unclear.^2-7

In EBM education, there is a divide between those who propone information mastery versus EBM expert models.8,9 The expert model emphasizes critical appraisal skills; instruction is provided in journal clubs outside of clinical activities and focuses on analysis of individual studies. Information mastery curricula emphasize synthesized evidence to inform patient care decisions. While journal clubs were the nearly sole method of teaching EBM before 2000, research supports an integrated, comprehensive EBM curriculum, as the most effective to achieve behavior change.2,12,13 Although learners may prefer the expert model,9 others8,14,15 argue that EBM education should focus on information mastery.

The University of Rochester Family Medicine Residency employs a clinically-integrated12 EBM curriculum, entitled “Science of Family Medicine” (SOFM), that strives for balance between the two approaches. We agree that information mastery is the primary way for family physicians to provide timely, evidence-based care.8,15 We also believe that rehearsal of expert EBM skills makes one a more competent information master, and facilitates an enhanced understanding of the synthesized resources commonly accessed at the point of care. With few exceptions,16-19 most other curricula focus on one approach or the other, with the majority still emphasizing critical appraisal.2,6,8,12, 20-31 In contrast, SOFM requires acumen with both point-of-care synthesized resources and critical appraisal.

SOFM is cyclical, with each third-year resident leading three sessions over 4 weeks, for a total of 11-13 cycles per year (Figure 1). Session 1 focuses on information mastery. Residents practice critical appraisal during session 2. The final presentation is a peer teaching session encompassing content from sessions 1 and 2. We developed a course website and integrated a medical librarian (L.P.) and an epidemiologist (B.A.L.). Table 1 lists curricular goals. Prior to the COVID-19 pandemic, sessions 1 and 2 occurred at a local diner, incentivizing engagement. We currently meet virtually.

One barrier to EBM practice can be institutional culture.6,32,33 We aimed to evaluate SOFM’s effect on resident perception of our EBM environment. The Evidence-Based Medicine Environment Survey (EBMES)32 provided a validated instrument for this purpose. It contrasts with other EBM assessments that focus on learners rather than curricula or environment.3,34-36 We hypothesized that resident perception of our EBM learning environment would improve after SOFM.

The EBMES is a 36-item survey evaluating contextual factors affecting EBM practice on a 5-point Likert scale. Rigorous development and psychometrics give evidence for content, internal structure, and discriminant validities, and protect against acquiescence bias.32 We obtained the survey from the authors. We distributed the EBMES to two cohorts of third-year family medicine residents before and after SOFM. The 2019-2020 academic year is cohort 1 and 2020-2021 is cohort 2. EBMES completion was neither required nor tied to evaluation.

We pulled, matched, and de-identified data before analyses. We added cohorts’ 1 and 2 pre- and posttest data together. We calculated mean values for pre- and posttests and conducted paired t tests to determine the difference between pretest and posttest for the combined cohort and stratified by cohort using Stata software, version 16 (StataCorp, LLC). Per the University of Rochester’s Research Subjects Review Board, this evaluation (STUDY00004483) did not meet the federal definition of research.

Twenty-one PGY-3 residents completed both pre- and posttests. From cohort 1, 11/11 residents completed the pretest, and 10/11 completed the posttest. From cohort 2, 11/12 residents completed the pretest, and 12/12 completed the posttest. Table 2 shows mean pre- and postscores for each EBMES item, with each item moving in a favorable direction, and significant differences between pre- and posttest shown for 36.1% (n=13/36) in cohort 1, 25.0% (n=9/36) in cohort 2, and 41.7% (n=15/36) in the combined cohort. Almost all items which were not statistically significant in the combined cohort were not directly covered in the curriculum, had a favorable baseline with little room for improvement (ie, >4 or <2), or met both of these criteria. Some items, despite being statistically significant in the combined cohort, did not have statistically significant improvements when examining individual cohorts due to small sample sizes. All but two of the 15 statistically significant items are directly covered in SOFM. Additionally, all but two of the 21 items without statistically significant pre/post differences were not covered in our curriculum. In the combined cohort, respondents had the highest posttest agreement with “value of adopting EBM in my clinical practice.” The item with lowest agreement in both cohorts was “attending physician provides me with clear feedback on my EBM practice.”

Nearly 100% of content covered in SOFM and measured by the EBMES favorably changed in a statistically significant manner. Some items, although not directly covered, may be influenced by our curriculum. Our next step is to improve faculty feedback around resident EBM practice.

SOFM balances expert EBM skills with information mastery. Our EBM environment was well perceived in the preintervention survey, with all items scored better than neutral. Despite this high baseline, 41.9% of items had significant improvement, which may be an effect of the SOFM curriculum or of confounding factors. Our curriculum adheres to recommendations for EBM education best practices,2,12,13,37,38,20 which may contribute to positive effect.

By using the EBMES, our goal was to evaluate curriculum, not learners. We found no implementation barriers specific to the tool. This is the first application of the EBMES as a curricular evaluation tool.

Our evaluation has several limitations. Educationally meaningful differences between pre- and posttests are not defined. There are no minimum thresholds indicating an adequate EBM learning environment. Amount of training and trainee level may be at least as important as content and structure.13,32 Culture may not be the primary barrier to implementation of EBM.39-42

We are encouraged by improvements in resident perception of our EBM learning environment after SOFM. Further research should examine pairing of information mastery with EBM expert training. Until then, SOFM is one possible model for family medicine residency EBM education. Its key components, inclusive of recurring cycles, peer teaching, information mastery, critical appraisal, and a social atmosphere with food when possible, can be modified to fit other graduate family medicine contexts.