— BRIEF REPORTS —

Description and Evaluation of an Evidence-Based Residency Curriculum Using the Evidence-Based Medicine Environment Survey

Celeste Song, MD, MS | Lorraine Porcello, MSLIS, MSIM | Tresne Hernandez | Brooke A. Levandowski, PhD, MPA

Fam Med. 2022;54(4):298-303.

DOI: 10.22454/FamMed.2022.652106

Abstract

Background and Objectives: Teaching medical trainees evidence-based medicine (EBM) is required by the Accreditation Council for Graduate Medical Education. Most published graduate EBM curricula focus on critical appraisal over point-of-care information mastery. Faculty at the University of Rochester Family Medicine Residency implemented a clinically integrated, cyclical EBM curriculum juxtaposing information mastery with expert-level skills such as critical appraisal. We administered the Evidence-Based Medicine Environment Survey (EBMES) to learners before and after the yearlong curriculum.

Methods: Two cohorts of participating third-year residents completed the EBMES before and after an EBM curriculum.

Results: Over 2 years, 21 residents completed pre- and postevaluations. Resident perception of the EBM educational and practice environment was high at baseline and improved for 15 of 36 survey items (P<.05).

Conclusions: Resident perception of the EBM learning environment improved after participation in a yearlong curriculum. Nearly all of the content covered in the "Science of Family Medicine" curriculum and measured by EBMES improved in a statistically significant manner. We propose that EBM curricula should combine traditional literature search and critical appraisal skills with information mastery to maximize effectiveness. Our curriculum can be modified to fit other graduate family medicine contexts.


Evidence-based medicine (EBM) education is required by the Accreditation Council for Graduate Medical Education.1 Practice-Based Learning and Improvement-1 encapsulates EBM competencies and is harmonized across specialties.1 Best practices for EBM curricular content and methods are unclear.2-7

In EBM education, there is a divide between those who propone information mastery versus EBM expert models.8,9 The expert model emphasizes critical appraisal skills; instruction is provided in journal clubs outside of clinical activities and focuses on analysis of individual studies. Information mastery curricula emphasize synthesized evidence to inform patient care decisions. While journal clubs were the nearly sole method of teaching EBM before 2000, research supports an integrated, comprehensive EBM curriculum, as the most effective to achieve behavior change.2,12,13 Although learners may prefer the expert model,9 others8,14,15 argue that EBM education should focus on information mastery.

The University of Rochester Family Medicine Residency employs a clinically-integrated12 EBM curriculum, entitled “Science of Family Medicine” (SOFM), that strives for balance between the two approaches. We agree that information mastery is the primary way for family physicians to provide timely, evidence-based care.8,15 We also believe that rehearsal of expert EBM skills makes one a more competent information master, and facilitates an enhanced understanding of the synthesized resources commonly accessed at the point of care. With few exceptions,16-19 most other curricula focus on one approach or the other, with the majority still emphasizing critical appraisal.2,6,8,12, 20-31 In contrast, SOFM requires acumen with both point-of-care synthesized resources and critical appraisal.

Curriculum Structure

SOFM is cyclical, with each third-year resident leading three sessions over 4 weeks, for a total of 11-13 cycles per year (Figure 1). Session 1 focuses on information mastery. Residents practice critical appraisal during session 2. The final presentation is a peer teaching session encompassing content from sessions 1 and 2. We developed a course website and integrated a medical librarian (L.P.) and an epidemiologist (B.A.L.). Table 1 lists curricular goals. Prior to the COVID-19 pandemic, sessions 1 and 2 occurred at a local diner, incentivizing engagement. We currently meet virtually.

Evaluation

One barrier to EBM practice can be institutional culture.6,32,33 We aimed to evaluate SOFM’s effect on resident perception of our EBM environment. The Evidence-Based Medicine Environment Survey (EBMES)32 provided a validated instrument for this purpose. It contrasts with other EBM assessments that focus on learners rather than curricula or environment.3,34-36 We hypothesized that resident perception of our EBM learning environment would improve after SOFM.

Methods

The EBMES is a 36-item survey evaluating contextual factors affecting EBM practice on a 5-point Likert scale. Rigorous development and psychometrics give evidence for content, internal structure, and discriminant validities, and protect against acquiescence bias.32 We obtained the survey from the authors. We distributed the EBMES to two cohorts of third-year family medicine residents before and after SOFM. The 2019-2020 academic year is cohort 1 and 2020-2021 is cohort 2. EBMES completion was neither required nor tied to evaluation.

We pulled, matched, and de-identified data before analyses. We added cohorts’ 1 and 2 pre- and posttest data together. We calculated mean values for pre- and posttests and conducted paired t tests to determine the difference between pretest and posttest for the combined cohort and stratified by cohort using Stata software, version 16 (StataCorp, LLC). Per the University of Rochester’s Research Subjects Review Board, this evaluation (STUDY00004483) did not meet the federal definition of research.

Results

Twenty-one PGY-3 residents completed both pre- and posttests. From cohort 1, 11/11 residents completed the pretest, and 10/11 completed the posttest. From cohort 2, 11/12 residents completed the pretest, and 12/12 completed the posttest. Table 2 shows mean pre- and postscores for each EBMES item, with each item moving in a favorable direction, and significant differences between pre- and posttest shown for 36.1% (n=13/36) in cohort 1, 25.0% (n=9/36) in cohort 2, and 41.7% (n=15/36) in the combined cohort. Almost all items which were not statistically significant in the combined cohort were not directly covered in the curriculum, had a favorable baseline with little room for improvement (ie, >4 or <2), or met both of these criteria. Some items, despite being statistically significant in the combined cohort, did not have statistically significant improvements when examining individual cohorts due to small sample sizes. All but two of the 15 statistically significant items are directly covered in SOFM. Additionally, all but two of the 21 items without statistically significant pre/post differences were not covered in our curriculum. In the combined cohort, respondents had the highest posttest agreement with “value of adopting EBM in my clinical practice.” The item with lowest agreement in both cohorts was “attending physician provides me with clear feedback on my EBM practice.”

Discussion

Nearly 100% of content covered in SOFM and measured by the EBMES favorably changed in a statistically significant manner. Some items, although not directly covered, may be influenced by our curriculum. Our next step is to improve faculty feedback around resident EBM practice.

SOFM balances expert EBM skills with information mastery. Our EBM environment was well perceived in the preintervention survey, with all items scored better than neutral. Despite this high baseline, 41.9% of items had significant improvement, which may be an effect of the SOFM curriculum or of confounding factors. Our curriculum adheres to recommendations for EBM education best practices,2,12,13,37,38,20 which may contribute to positive effect.

By using the EBMES, our goal was to evaluate curriculum, not learners. We found no implementation barriers specific to the tool. This is the first application of the EBMES as a curricular evaluation tool.

Our evaluation has several limitations. Educationally meaningful differences between pre- and posttests are not defined. There are no minimum thresholds indicating an adequate EBM learning environment. Amount of training and trainee level may be at least as important as content and structure.13,32 Culture may not be the primary barrier to implementation of EBM.39-42

We are encouraged by improvements in resident perception of our EBM learning environment after SOFM. Further research should examine pairing of information mastery with EBM expert training. Until then, SOFM is one possible model for family medicine residency EBM education. Its key components, inclusive of recurring cycles, peer teaching, information mastery, critical appraisal, and a social atmosphere with food when possible, can be modified to fit other graduate family medicine contexts.

Acknowledgments

The authors thank research assistant Nancy Phillips, who pulled, matched, and deidentified all study data.

Funding Statement: Dr Song receives salary support for directing the Science of Family Medicine Curriculum. 

References

  1. Accreditation Council for Graduate Medical Education; American Board of Family Medicine. The Family Medicine Milestone Project. Updated 2019. Accessed May 12, 2020. https://www.acgme.org/Portals/0/PDFs/Milestones/FamilyMedicineMilestones2.0.pdf?ver=2019-10-21-152802-123
  2. Ilic D, Maloney S. Methods of teaching medical trainees evidence-based medicine: a systematic review. Med Educ. 2014;48(2):124-135. doi:10.1111/medu.12288
  3. Flores-Mateo G, Argimon JM. Evidence based practice in postgraduate healthcare education: A systematic review. BMC Health Serv Res. 2007;7:119-6963-7-119. [pii]. doi:10.1186/1472-6963-7-119
  4. Ahmadi N, McKenzie ME, Maclean A, Brown CJ, Mastracci T, McLeod RS; Evidence-Based Reviews in Surgery Steering Group. Teaching evidence based medicine to surgery residents-is journal club the best format? A systematic review of the literature. J Surg Educ. 2012;69(1):91-100. doi:10.1016/j.jsurg.2011.07.004
  5. Ilic D, de Voogt A, Oldroyd J. The use of journal clubs to teach evidence-based medicine to health professionals: A systematic review and meta-analysis. J Evid Based Med. 2020;13(1):42-56. doi:10.1111/jebm.12370
  6. Simons MR, Zurynski Y, Cullis J, Morgan MK, Davidson AS. Does evidence-based medicine training improve doctors’ knowledge, practice and patient outcomes? A systematic review of the evidence. Med Teach. 2019;41(5):532-538. doi:10.1080/0142159X.2018.1503646
  7. Hecht L, Buhse S, Meyer G. Effectiveness of training in evidence-based medicine skills for healthcare professionals: A systematic review. BMC Med Educ. 2016;16:103-016-0616-2. doi:10.1186/s12909-016-0616-2
  8. Shaughnessy AF, Gupta PS, Erlich DR, Slawson DC. Ability of an information mastery curriculum to improve residents’ skills and attitudes. Fam Med. 2012;44(4):259-264.
  9. Akl EA, Maroun N, Neagoe G, Guyatt G, Schunemann HJ. EBM user and practitioner models for graduate medical education: What do residents prefer? Med Teach. 2006;28(2):192-194. doi: J7N2V46339726VLH [pii].
  10. Green ML. Evidence-based medicine training in graduate medical education: past, present and future. J Eval Clin Pract. 2000;6(2):121-138. doi:10.1046/j.1365-2753.2000.00239.x
  11. Hatala R, Keitz SA, Wilson MC, Guyatt G. Beyond journal clubs. Moving toward an integrated evidence-based medicine curriculum. J Gen Intern Med. 2006;21(5):538-541. doi:10.1111/j.1525-1497.2006.00445.x
  12. Coomarasamy A, Khan KS. What is the evidence that postgraduate teaching in evidence based medicine changes anything? A systematic review. BMJ. 2004;329(7473):1017. doi:10.1136/bmj.329.7473.1017
  13. Ilic D, Diug B. The impact of clinical maturity on competency in evidence-based medicine: a mixed-methods study. Postgrad Med J. 2016;92(1091):506-509. doi:10.1136/postgradmedj-2015-133487. doi.
  14. Slawson DC, Shaughnessy AF. Teaching evidence-based medicine: should we be teaching information management instead? Acad Med. 2005;80(7):685-689. doi:10.1097/00001888-200507000-00014
  15. Ebell MH. AFP’s series on finding evidence and putting it into practice. Am Fam Physician. 2009;79(1):7-8.
  16. Allan GM, Korownyk C, Tan A, Hindle H, Kung L, Manca D. Developing an integrated evidence-based medicine curriculum for family medicine residency at the University of Alberta. Acad Med. 2008;83(6):581-587. doi:10.1097/ACM.0b013e3181723a5c
  17. Bradley DR, Rana GK, Lypson ML, Hamstra SJ. A centralized practice-based learning and improvement curriculum for residents and fellows: a collaboration of health sciences librarians and graduate medical education administration. J Med Libr Assoc. 2010;98(2):175-178. doi:10.3163/1536-5050.98.2.013. doi.
  18. Bounds R, Boone S. The flipped journal club. West J Emerg Med. 2018;19(1):23-27. doi:10.5811/westjem.2017.11.34465
  19. Nandiwada DR, Kohli A, McNamara M, et al. High-value consults: A curriculum to promote point-of-care, evidence-based recommendations. J Grad Med Educ. 2017;9(5):640-644. doi:10.4300/JGME-D-17-00177.1
  20. Chitkara MB, Boykan R, Messina CR. A longitudinal practical evidence-based medicine curriculum for pediatric residents. Acad Pediatr. 2016;16(3):305-307. doi:10.1016/j.acap.2015.12.013. doi.
  21. Coomarasamy A, Taylor R, Khan KS. A systematic review of postgraduate teaching in evidence-based medicine and critical appraisal. Med Teach. 2003;25(1):77-81. doi:10.1080/0142159021000061468. doi.
  22. Aneese AM, Nasr JA, Halalau A. A prospective mixed-methods study evaluating the integration of an evidence based medicine curriculum into an internal medicine residency program. Adv Med Educ Pract. 2019;10:533-546. doi:10.2147/AMEP.S203334
  23. Mohr NM, Stoltze AJ, Harland KK, Van Heukelom JN, Hogrefe CP, Ahmed A. An evidence-based medicine curriculum implemented in journal club improves resident performance on the Fresno test. J Emerg Med. 2015;48(2):222-229.e1. doi:10.1016/j.jemermed.2014.09.011
  24. Kenefick CM, Boykan R, Chitkara M. Partnering with residents for evidence-based practice. Med Ref Serv Q. 2013;32(4):385-395. doi:10.1080/02763869.2013.837669
  25. Kim S, Willett LR, Murphy DJ, O’Rourke K, Sharma R, Shea JA. Impact of an evidence-based medicine curriculum on resident use of electronic resources: a randomized controlled study. J Gen Intern Med. 2008;23(11):1804-1808. doi:10.1007/s11606-008-0766-y
  26. Kulier R, Hadley J, Weinbrenner S, et al. Harmonising evidence-based medicine teaching: A study of the outcomes of e-learning in five european countries. BMC Med Educ. 2008;8:27-6920-8-27. doi:10.1186/1472-6920-8-27
  27. Epling J, Smucny J, Patil A, Tudiver F. Teaching evidence-based medicine skills through a residency-developed guideline. Fam Med. 2002;34(9):646-648.
  28. Rohwer A, Young T, van Schalkwyk S. Effective or just practical? An evaluation of an online postgraduate module on evidence-based medicine (EBM). BMC Med Educ. 2013;13:77-6920-13-77.
  29. Nasr JA, Falatko J, Halalau A. The impact of critical appraisal workshops on residents’ evidence based medicine skills and knowledge. Adv Med Educ Pract. 2018;9:267-272. doi:10.2147/AMEP.S155676
  30. Nelson B, Ingard C, Nelson D. Teaching trainees how to critically evaluate the literature - a crossover study at two pediatric residency programs. Int J Med Educ. 2017;8:137-141. doi:10.5116/ijme.58ce.5f04
  31. Chitkara M, Boykan R. An integrated practical evidence-based medicine curriculum, critical appraisal: systematic reviews and meta-analyses. MedEdPORTAL. 2014;2374. doi:10.15766/mep_2374-8265.9835
  32. Mi M, Moseley JL, Green ML. An instrument to characterize the environment for residents’ evidence-based medicine learning and practice. Fam Med. 2012;44(2):98-104.
  33. Vidyarthi AR, Kamei R, Chan K, Goh SH, Lek N. Factors associated with medical student clinical reasoning and evidence based medicine practice. Int J Med Educ. 2015;6:142-148. doi:10.5116/ijme.563a.5dd0
  34. Shaneyfelt T, Baum KD, Bell D, et al. Instruments for evaluating education in evidence-based practice: a systematic review. JAMA. 2006;296(9):1116-1127. doi:10.1001/jama.296.9.1116
  35. Thomas RE, Kreptul D. Systematic review of evidence-based medicine tests for family physician residents. Fam Med. 2015;47(2):101-117.
  36. Kumaravel B, Hearn JH, Jahangiri L, Pollard R, Stocker CJ, Nunan D. A systematic review and taxonomy of tools for evaluating evidence-based medicine teaching in medical education. Syst Rev. 2020;9(1):91-020-01311-y. doi:10.1186/s13643-020-01311-y
  37. Das K, Malick S, Khan KS. Tips for teaching evidence-based medicine in a clinical setting: lessons from adult learning theory. Part one. J R Soc Med. 2008;101(10):493-500. doi:10.1258/jrsm.2008.080712. doi.
  38. Malick S, Das K, Khan KS. Tips for teaching evidence-based medicine in a clinical setting: lessons from adult learning theory. Part two. J R Soc Med. 2008;101(11):536-543. doi:10.1258/jrsm.2008.080713
  39. Barzkar F, Baradaran HR, Koohpayehzadeh J. Knowledge, attitudes and practice of physicians toward evidence-based medicine: A systematic review. J Evid Based Med. 2018;11(4):246-251. doi:10.1111/jebm.12325
  40. Pather MK, Mash R. Family physicians’ experience and understanding of evidence-based practice and guideline implementation in primary care practice, Cape Town, South Africa. Afr J Prim Health Care Fam Med. 2019;11(1):e1-e10. doi:10.4102/phcfm.v11i1.1592
  41. Paulsen J, Al Achkar M. Factors associated with practicing evidence-based medicine: a study of family medicine residents. Adv Med Educ Pract. 2018;9:287-293. doi:10.2147/AMEP.S157792
  42. Mi M, Halalau A. A pilot study exploring the relationship between lifelong learning and factors associated with evidence-based medicine. Int J Med Educ. 2016;7:214-219. doi:10.5116/ijme.576f.a2ca

Lead Author

Celeste Song, MD, MS

Affiliations: Department of Family Medicine, University of Rochester School of Medicine and Dentistry, Rochester, NY

Co-Authors

Lorraine Porcello, MSLIS, MSIM - Institute for Innovative Education, University of Rochester School of Medicine and Dentistry, Rochester, NY

Tresne Hernandez - University of Rochester School of Medicine and Dentistry, Rochester, NY

Brooke A. Levandowski, PhD, MPA - Department of Obstetrics and Gynecology, University of Rochester School of Medicine and Dentistry, Rochester, NY

Fetching other articles...

Mendeley

Loading metrics from Mendeley...

Loading the comment form...

Submitting your comment...

There are no comments for this article.

Downloads & Info

Share

Tags

Searching for articles...