Examining an Evidence-Based Medicine Culture in Residency Education

Training in evidence-based medicine (EBM) skills, such as critical appraisal of medical research literature and its judicious, conscientious, and explicit application to clinical practice,1 has been an integral component of family medicine residency education since the early 2000s. Its importance is reflected primarily in the Accreditation Council for Graduate Medical Education’s (ACGME’s) Problem Based Learning and Improvement competency and related milestones and some of the related, newly-developed entrustable professional activities (EPAs). The existing educational literature on this topic has mainly focused on describing the variety and content of EBM curricula, and on assessment of knowledge and skills related to the practice of EBM.2-4 Several of this paper’s authors (J.E., J.H., D.W., G.C.)—members of the Society of Teachers of Family Medicine (STFM) Group on Evidence-Based Medicine—reflected on the status of EBM teaching in medical school and in family medicine residency programs. From our collective anecdotal experience, we noted that in many programs this teaching is relegated to a small minority of faculty members in a school or program who possess expertise and interest in this area.

Research into either specifically designated or hidden curricula in US medical school training reveals that the culture of an educational program can either reinforce or subvert curricular initiatives.5 We believe role modeling and a supportive training environment are essential components in all aspects of medical school education and residency training. However, to date, these aspects of training in EBM have not been well explored in the educational literature. A literature review revealed several cultural factors that could influence the success of EBM training curricula, such as authentic curricular exercises, role modeling, and sufficient resources.3,6-18

The best quality work in this area was a survey study by Mi, et al that examined the EBM training environment from a learner’s perspective.19,20 Extensive survey validation was performed, and the results showed that a supportive EBM learning culture, as perceived by the trainees, was important in EBM education. We recognized the opportunity to complement this research with a nationwide survey of residency program directors in family medicine, using a modified form of the Mi, et al survey, in order to assess the extent to which programs possessed the elements of this supportive EBM culture.

We adapted the validated survey from Mi, et al containing 36 items in seven subscales for use in the Council of Academic Family Medicine Educational Research Alliance’s (CERA’s) Program Directors Survey, an omnibus survey conducted annually.21 We eliminated items from two subscales of Mi’s EBM scale that focused solely on learner attitudes and accountability. The remaining subscales focused on resource availability, social support, learning support, situational cues, and learning culture. Because of the limitations on survey length in CERA surveys, we trimmed duplicative items from the Mi survey, but retained key questions from each subscale. We incorporated the edited subscales of 13 items into the final CERA survey. These items, which formed the EBM Culture Scale (ECS), asked respondents to rate their agreement with the item statements on a 5-point Likert scale from 1 (strongly disagree) to 5 (strongly agree). The survey was pretested to ensure clarity of the questions with family medicine educators who were not part of the target survey population. The American Academy of Family Physicians Institutional Review Board approved the project in January 2015. The survey (deployed using SurveyMonkey22) was sent by email to all US family medicine residency program directors identified by the Association of Family Medicine Residency Directors. Data was collected from January 2015 to March of 2015 using email invitations to the SurveyMonkey platform. Two follow-up reminders were sent to enhance response rate.

We analyzed the survey results descriptively for response rate, demographic information of the respondents, and information about the programs. We analyzed survey item internal consistency using Cronbach α. We examined the frequencies and means of the responses to the ECS items and created an overall ECS score by summing the scores for the 13 items. We further analyzed the results for this overall score by conducting a linear regression on the ECS score by examining, as coefficients, the demographics obtained as part of the larger survey that were significantly associated with the EBM score in a priori testing. All analyses were conducted using R version 3.2.2, with the psych package.23,24

We received 274 responses out of a sampled population of 452 (60.6% response rate). The respondents were mostly male (64%) and had a median of 4 years of experience as program director (interquartile range 1.5 to 7.5 years). The mean age of the respondents’ residency programs was 33.1 years (SD 13.3 years); further program characteristics are shown in Table 1. The survey showed a good level of internal consistency (Cronbach α of 0.85 [95% CI 0.81-0.89]).

The mean score of each ECS item is shown in Table 2. The most strongly-endorsed items included the practice environment’s commitment to lifelong learning, the program’s cultivation of an atmosphere of mutual respect, and the encouragement of residents as problem solvers. The least strongly-endorsed items included faculty feedback to residents about EBM skills, faculty involvement in teaching EBM, and protection of resident time for EBM training. The mean overall ECS score was 56.3 (SD 5.64) out of a maximum score of 65, and most program directors scored their programs greater than 50.

We explored the association of overall ECS score with each of the demographic items as a priori testing for our linear regression analysis. Only the gender of the program director (t=2.47, P=0.01) and proportion of international medical graduates (IMGs) in the program (ANOVA, F=2.43, P=0.048) were significantly correlated with the overall ECS score. Further a priori testing revealed a bifurcation of the proportion of international graduates variable at 50%, and we created a corresponding dichotomous variable for use in the analysis. We performed a linear regression analysis of the overall ECS score using program director gender and proportion of IMGs over 50% (Table 3). Female gender of the program director predicted a 1.88-point decrease (95% CI 0.41 to 3.34 points) in the overall ECS score. Despite some correlation seen in a priori testing between program director gender and years of experience (P=0.003), adding years as program director did not contribute significantly to the model. Proportion of IMGs over 50% in the program predicted a 2.23-point decrease (95% CI 0.71 to 3.75) in the ECS score.

Overall, this survey provides thought-provoking data about program directors’ self-assessment of their programs’ learning environments for teaching EBM. Most program directors rated their programs high on the EBM scale, which may result from both a global expectation (from the ACGME and others) that residencies teach EBM well, as well as a natural tendency for positive self-assessment. Examining the individual items, it seems that both the involvement of faculty in EBM teaching and their delivery of feedback to the residents about their EBM skills present the most challenges. This could be due to the time available to deliver this feedback, the faculty’s self-efficacy in EBM skills, the faculty’s self-efficacy in teaching and assessing EBM skills, or the perceived effectiveness of this feedback. Program directors generally rated the atmosphere and environmental statements better than the specifics of EBM teaching or the EBM-related teaching infrastructure.

The difference in the prediction of overall EBM score by program director gender could be explained by a documented tendency of women to self-evaluate more harshly than men concerning managerial and performance-related domains.25,26 There is no obvious mechanism by which this prediction would represent a true difference. For the lower EBM scores associated with a higher proportion of IMGs, some research has suggested that a lack of EBM education in medical school may underlie the relationship.27 In addition, smaller and less-established residency programs frequently have a high proportion of IMGs, so we postulate that there may be less overall curricular infrastructure in these programs. There may also be greater curricular attention to acculturation and communication issues in these residencies, thereby displacing other curricular elements. This finding deserves more study examining both factual curriculum components and perceptions or attitudes toward teaching of EBM-integrated knowledge.

This study is limited by the adaptation of a previously validated survey for use with a different respondent population. In order to incorporate the ECS scale within the omnibus CERA survey, the original scale developed and validated by Mi was modified by the elimination of a number of items. As expected, the original factor structure identified by Mi was not preserved in the modified instrument. However, all remaining items were closely associated with one another, and the most efficient analytic step was to utilize the overall ECS score in the regression analysis. It is possible that use of the original, complete ECS scale as validated by Mi would have yielded different results. However, we feel that the careful selection of the applicable factors from the original Mi survey and the thematically consistent results we obtained enable useful conclusions to be drawn.

Another limitation is the highly subjective self-assessment of residency program characteristics, that depends on the respondents’ ability to meaningfully compare their program to a larger and nondefined standard. We attempted to collect data about the number of residency slots per year in each program, but the data were clearly unreliable, as it appeared that many respondents answered relating the total available slots in the residency program, and we did not have an acceptable method for distinguishing the errors from actual data.

The strengths of this study include the overall response rate, and the adaptation of a validated survey. Because of the geographically broad representation of the survey respondents, community size and type of program (see Table 1), and its reasonably high response rate of 60.6%, these findings can reasonably generalize to all US family medicine residencies.

The findings identify important EBM contextual factors that were confirmed in another recent study by Bergh, et al: knowledge and learning materials; learner support; general relationships and support; institutional focus on EBM; education, training, and supervision; EBM application opportunities; and affirmation of EBM environment.28 Our findings are also in accordance with those of several other studies indicating the importance of these environmental/contextual factors that contribute to the culture of promoting or sustaining learning and adoption of EBM. Specifically, those studies noted the importance of faculty role modeling, time dedicated to the EBM curriculum, institutional support, collaboration from a library and similar nonmedical faculty, and leadership commitment.29-31

The results of this study suggest strong implications for residency programs in developing and implementing an EBM intervention. Effective learning outcomes and application of EBM knowledge and skills are associated with learner factors (eg, prior EBM training and experience) and learning support through faculty feedback, protected time, role modelling, or mentoring. Based on the survey results, we suggest that to improve the delivery of an EBM curriculum in residency, program directors should focus on authentic activity (eg, integrating and reinforcing EBM teaching in the clinical environment), whole-residency involvement and commitment, and a suitable general and EBM-specific infrastructure for learning and application of learning. Future research is warranted to investigate the impact of affective and environment factors on EBM learning and practice, so EBM training can be designed and implemented to optimize EBM training outcomes to improve patient care and promote residents’ critical thinking, problem-solving, and lifelong learning.