Evidence-Based Medicine Culture, Curriculum, and Program Outcomes: A CERA Study

Evidence-based medicine (EBM) is the application of the best available research to the care of an individual patient. Since the 1990s, EBM has been conceived and taught as critical appraisal of original research for the purpose of answering a clinical point-of-care question. ¹ The concept has evolved and expanded to include lifelong learning, information management, and evidence synthesis. EBM is an integral part of family medicine ideals: shared decision-making and value-based care. Shared decision-making requires sufficient knowledge of high-quality evidence as well as skill to counsel a patient on the risks and benefits for their individual situation. Value-based care relies on evidence to support its emphasis on cost-effectiveness. Despite these aspirational ideals, evidence has suggested that family physicians experience substantial barriers to teaching, learning, and exercising evidence-based practices. ^2-5

The best methods for teaching EBM are not clear. Previous studies were limited by changing resources, evolving technology, and methodologic constructs over the past 20 years.^6- 8 Traditionally, many family medicine residency programs have relied on curricula emphasizing only one facet of EBM, such as point-of-care questions and answers or critical appraisal of original research. ⁹ Longitudinal curricula with varying degrees of clinical integration have been studied, but whether they are more effective at producing family medicine residents with confidence and skills at incorporating EBM into practice is unclear. ^9- 12 Few of the existing studies have included behavioral outcomes (ie, outcomes other than knowledge or skill). In addition to didactic curricula, practical experience with advanced EBM skills such as research, authoring, or previous EBM training may be associated with increased EBM skills.^13-15 Although critical, this experience, particularly research interest, is negatively correlated with an interest in family medicine. ¹⁶

Research conducted in 2015 demonstrated that family medicine residencies value evidence-based practice, and family medicine program directors reported a strong culture of EBM.¹⁷ However, a lack of opportunities for faculty development has been shown to be a limitation to advancing EBM education. ¹⁸ Faculty skill, time for EBM teaching, and, in some cases, a perceived tension between patient-centered care and evidence-based care, are other identified barriers.^{17, 19} Literature also has shown a lack of change in self-reported EBM skills during training among family medicine residents. ¹⁴

This study sought to describe family medicine program director perceptions of EBM and to report on EBM practices in family medicine residency training. We further sought to assess the association among structured faculty roles and EBM curricula to specific resident outcomes such as resident ability to identify errors in research and resident preparation to author review articles.

Between April 18 and May 12, 2023, family medicine program directors of US programs accredited by the Accreditation Council for Graduate Medical Education (ACGME) were invited to participate in the Council of Academic Family Medicine Educational Research Alliance (CERA) study. ²⁰ The survey was sent electronically to all family medicine program directors who had not previously opted out. Escalating reminders were sent for a total of five invitations over 4 weeks.

The survey questions were developed by members of the research team based on hypotheses, expert opinion, and literature review. They were reviewed by all members of the research team and revised iteratively to achieve the 10 questions allowed on the omnibus survey. Items assessed family medicine program directors’ perceptions of EBM curriculum, EBM faculty experience and expertise, EBM program culture, and resident EBM outcomes. Program directors reported on faculty expertise (novice to national recognition) and faculty years of experience with EBM.

The CERA steering committee evaluated all questions for reliability and validity based on evidence presented. A sample of family medicine educators who were not part of the survey pretested the questions.

We analyzed survey data with Statistical Analysis Software version 9.4 (SAS Institute) using descriptive statistics and independent ꭓ² tests for associations. Missing responses were excluded from analyses, and valid percentages were reported. Response options were collapsed for independent ꭓ² analyses due to the small numbers of responses to several options.

The project was approved by the American Academy of Family Physicians Institutional Review Board in April 2023.

The overall response rate for the survey was 44.72% (309/691; Table 1).

A total of 42.0% (118/281) of program directors reported that EBM was formalized into both didactics and clinical experiences. And 37.0% (104/281) reported that EBM was formalized into didactics only: 24.2% into didactics beyond journal club and 12.8% into journal club only (Table 2).

More than 90% of program directors agree/strongly agree that both faculty (255/281; 90.7%) and residents (253/280; 90.3%) accept the process and outcome of an evidence search for answers to clinical questions. Approximately 83% (234/281) disagree/strongly disagree that clinicians find reasons to doubt or reject evidence, or avoid incorporating it into practice.

Of the 309 program directors, 264 (84.9%) reported that fewer than 50% of the residents in their program would be able to identify a major error or omission in an UpToDate article. And 143/309 (46.3%) reported that fewer than 25% would be able to do so.

We found that 214/309 (69.3%) of program directors reported that fewer than 50% of their residents would be able to identify a significant error in an original research study; 114/309 (36.9%) reported that fewer than 25% would be able to do so.

Twenty-one percent (61/309) of program directors reported that 0–1 of their current residents will graduate with the skills to be the lead author on a narrative review article. Ten percent (28/309) of program directors reported that at least 50% of their residents will be able to do so.

A total of 25.6% (72/309) of program directors reported that no identifiable person is leading an EBM curriculum at their program. Among the program directors, 23.6% reported that their EBM faculty has been in practice at least 16 years. Faculty experience greater than 16 years was associated with formalized EBM curriculum in both didactics and clinical experiences, or clinical experiences only (P<.001; Table 3).

We found a significant association between EBM faculty years in practice and expertise level of the faculty member (P=.0008). Program directors reported that 32% of EBM faculty members with 16 or more years of experience have national recognition compared to 18% of EBM faculty members with 11 to 15 years of experience, 17% with 5 to 10 years, and 11% with 0 to 4 years.

Only 31% of program directors reported that they could replace current EBM faculty from within their own faculty; 15.7% reported that it would take 12 months or more to do so.

The absence of a designated EBM faculty member was found to correlate with the absence of a formal EBM curriculum (P<.001). Of program directors that responded they had no specific EBM faculty lead, 22% (16/72) reported that they did not have a formal EBM curriculum, compared with 5/209 (2.4%) of programs with an EBM faculty lead.

Faculty years in practice were not associated with resident competency outcomes; more experienced faculty were not associated with improved residency outcomes (Table 4). The type of EBM curriculum (eg, formal or informal, didactic only or clinically integrated) was not associated with identifying omissions in UpToDate or authoring narrative reviews but was associated with identifying errors in original research (P=.0228; Table 5).

More than 90% of program directors strongly agreed or agreed that EBM is accepted by residents and faculty, and that their programs have an EBM curriculum. However, the reported acceptance of EBM and integration of EBM curricula are contrasted with program directors’ reporting of resident outcomes. Having program directors define the nature of their curriculum was not within the scope of our survey. Program directors reported that far fewer than one-half of current residents can identify a significant error in original research or a major omission in a resource such as UpToDate. These factors were included in the survey because they are consistent with expectations from the ACGME requirements, which call for residents to be able to “challenge the evidence” used to make decisions and to “understand the benefits and limitations of the medical literature.”²¹ Family physicians regularly rely on point-of-care and nonevidence-based tools such as UpToDate; yet, without the skills to identify important omissions or errors, we are dependent on the tools’ own assessment of trustworthiness and accuracy. This may negatively impact patient care.

This study showed that we are not graduating residents who are capable of identifying serious errors in research studies, can appropriately critique secondary sources, or can contribute to authoring evidence-based literature. Few program directors reported that their residents will graduate prepared to author an EBM-based narrative review article. Though family physicians rely on high-quality evidence-based reviews, residencies are not consistently training graduates to author, edit, and publish such works. A major aim in training family medicine residents is to have them learn from original research, ideally from EBM-trained experts through didactics, journal clubs, and patient encounters. This deficiency presents a missed opportunity for family physicians to become leaders and influencers among their peers, and it also raises concern about the development of current and future EBM faculty.

ACGME requires family medicine residents to demonstrate the ability to appraise and assimilate scientific evidence and to use that evidence to develop a patient care plan. A regular forum for discussing and analyzing evidence relevant to practice is also a core requirement. ²¹ Our study determined that about 25% of program directors reported having no specific faculty member responsible for their EBM curriculum. Programs without a faculty EBM lead were less likely to report having a formal EBM curriculum. Prior studies of EBM curricula have reported that limited time to teach EBM skills and difficulty recruiting EBM-skilled teachers are the largest barriers to implementing an EBM curriculum. ⁹ This study also found that program directors with more experienced EBM faculty members are more likely to report that their EBM curriculum is integrated into clinical practice. Taken together, these findings suggest that having designated faculty members and opportunities for ongoing faculty development are more likely to lead to integrated EBM curricula consistent with ACGME requirements.

Years of experience of the EBM lead faculty was not associated with any of the resident outcomes in this study. This finding suggests that more senior faculty are not more likely to produce better outcomes. Junior faculty can lead EBM curricula as successfully as senior faculty, although outcomes across the spectrum of faculty experience and expertise are lacking. As noted, the pathways for future designated EBM faculty development are of concern and are not well-defined. Only 11% of program directors reported having a faculty member with 0 to 4 years of experience responsible for their EBM curriculum. This finding, coupled with more than 26% of programs without a designated faculty member, suggests that significantly more resources and support for junior faculty and faculty development are needed. Clearly defined faculty development competencies would align family medicine EBM faculty around key areas of skill development. Similarly, the need to improve resident ability at critical appraisal of primary and secondary literature sources and evidence synthesis should drive the creation of resident-level competencies in EBM.

Our study demonstrated that while EBM skills are valued as necessary and integral to the practice of family medicine, most residency training programs lack sufficient faculty expertise and curricula to teach these skills in line with ACGME standards. This deficit ultimately will lead to continued reliance on nonevidence-based resources in clinical practice. Integration of robust EBM training is imperative to ensure that the upcoming generation of junior faculty will possess skills in EBM teaching, because our study found that nearly one-third of EBM-competent faculty have more than 16 years of faculty experience.

This was a self-report survey of program directors based on their perceptions of current faculty and expertise, and results may not correlate with the opinions of the EBM faculty or the residents themselves. Results may be subject to self-reporting biases, including social desirability bias and recall bias. The response rate of the survey was 44.7%, and we do not have information on nonresponders. Details regarding an EBM curriculum were not specifically defined, thus we assume moderate variability and heterogeneity across programs.

As a cross-sectional design, this study provides insight into a single point in time. Some analyses were affected by the need to group some responses due to the small number of responses to several options. In some cases, even after grouping, the validity of these analyses was still affected, and this should be taken into consideration when interpreting the findings.

Residency programs widely vary in cohort size and number of faculty, and some programs may have more than one identified EBM faculty member, which could include the program director. Our study was not designed to capture those data.

Family medicine residency program directors reported strong resident and faculty acceptance of EBM. However, they reported that few residents are graduating with EBM skills adequate for clinical practice. Many program directors reported not having an identified faculty member responsible for the EBM curriculum. Further study about effective teaching of EBM for residents and curriculum best practices is needed to foster integration of EBM resources and clinical practice. Development of continuing medical education and faculty development will help ensure a pipeline of effective future and current EBM faculty and will help meet ACGME core requirements for resident outcomes.

This study was presented at the 2024 Annual Meeting of the Society of Teachers of Family Medicine in Los Angeles, CA.