A Curriculum to Increase Resident Scholarly Activity

The Accreditation Council for Graduate Medical Education (ACGME) requires all programs to demonstrate evidence of scholarly activity.1 Refinement of resident requirements for scholarly activity are left to each specialty. Family medicine residents must participate in scholarship and should complete two scholarly activities prior to graduation, at least one of which should be a quality improvement project.2 To be recognized as scholarship, contributions must be shared with peers at a local or higher level and be subject to peer review.3 Widely recognized barriers to resident research include an absence or lack of protected time, access to mentors, a formal research curriculum, technical support, venues to present scholarship, and funding.4 Other barriers include a lack of interest5 and uncertainty.6 Myriad strategies have been developed within specific programs to overcome barriers to scholarly activity.7,8 Most of these strategies focus on overcoming the specific barriers listed above, and were well described by Seehusen and Weaver in 2009 across many programs,9 and Alweis et al in 2015 as iterative improvements within a single program.10

One successful alternative approach has been a scholarly activity point system, first described by Seehusen et al in 2009,11 and later shown to be successful at several other programs.12 Others include an improved journal club, a Wall of Fame, a writing seminar, and a cash incentive for residents.10 More recently, Seehusen et al have described the success of the Council of Academic Family Medicine Educational Research Alliance (CERA) efforts to build a research infrastructure for individuals external to their own programs in order to facilitate medical education research.13

While innovative and successful, we felt that greater change might be achieved if interventions were resident-driven, and developed a resident-driven curricular change at Naval Hospital Jacksonville Family Medicine Residency Program (NHJ). This resulted in a 13-fold increase in research activity and a seven-fold increase in resident participation in research.14 These improvements were sustained over the 5 years following implementation.15 In this study, we evaluate the outcomes after implementing this resident-driven curriculum at two additional family medicine residencies.

This was a prospective, multisite trial. The Naval Medical Center Portsmouth Institutional Review Board, deemed this project exempt, and it was funded through a grant from the Uniformed Services Academy of Family Physicians.

We distilled three core elements that led to NHJ’s success: a resident peer leader in research (the resident research coordinator, or RRC), a 3-hour case report workshop (CRW), and a guidebook to scholarly activity. The RRC is a self-selected resident who undertakes the position to champion research within their program, and is described in detail elsewhere.14,16 RRCs provide encouragement, direction, and support to residents pursuing scholarly activity. The CRW is a 3-hour, hands-on workshop available for use or adaptation from the Society of Teachers of Family Medicine Resource Library,17 and is described in detail elsewhere.14 The goal of the workshop is the creation of a near-complete abstract for submission. The guidebook used is The Recipe: A Practical Guide to Scholarly Activity, and was developed to provide residents with step-by-step instructions on scholarly activities from performing a literature search to conducting a process improvement project to writing a grant proposal. It is freely available for use online through both the Uniformed Services Academy of Family Physicians and the Society of Teachers of Family Medicine.16 Electronic versions of The Recipe were provided to all residents and faculty in fall of 2018.

Two full-scope ACGME-accredited military family medicine residency programs were invited to participate. Site A adopted all three intervention elements, identifying two residents to serve as research peer leaders, hosting a CRW and providing residents and faculty with The Recipe. Site B hosted the CRW and provided residents and faculty with The Recipe, but did not identify any residents to serve as research peer leaders.

Site A is a 12/12/12 program, and Site B is a 13/13/13 program. Each self-identified as a program desiring to increase their residents’ scholarly activity. While similarly sized military residencies, they are different at baseline in faculty research engagement and resident research time allotment. Several faculty at Site A are actively engaged in research; their residents receive no dedicated research time. Faculty at Site B are less actively engaged in research; their residents are given a 4-week block for research during their PGY-3 year.

It is challenging to define scholarly activity in a way that is rigorous, precise, generalizable, and timely. We have previously defined it as any activity generating scholarly activity points,14 or more rigorously, as presentation at a regional or higher conference or publication in a PubMed-indexed, peer-reviewed journal with a Medline number.15 To make measurements more generalizable, in this study we use data reported on the scholarly activity report by the program director (PD) to the ACGME. The reportable categories in the ACGME resident scholarly activity report include conference presentations, textbook chapters, research participation, teaching presentations and scholarly activity participation. Virtually all residents report teaching and some form of scholarly activity every year. For this reason, our primary outcome measures were limited to changes to conference presentation, textbook chapters and research participation. “Conference presentations” are abstracts, posters, or oral presentations given at regional or higher level conferences. “Research participation” includes participation in a funded or nonfunded basic science or clinical outcomes research project.

The PD at each site provided their ACGME-reported resident scholarly activity for academic years 2015-2016, 2016-17 and 2017-18 as baseline data. The intervention occurred in the 2018-19 academic year. In August 2019, the PD at each site provided their ACGME reported resident scholarly activity for the 2018-19 academic year. We compared pre- and postintervention ACGME-reported data using a 2-sample test for equality of proportions with continuity correction to 95% confidence, and confirmed with an alternate critical ratio test.

We collected resident perceptions about case reports before the CRW. We obtained these secondary outcomes to compare their perceptions of scholarly activity participation to their objective performance (measured as primary outcomes). Mismatches between resident perception of performance and objective measures of performance may highlight learning opportunities or knowledge gaps that hinder research. Residents were surveyed before the CRW, and resurveyed 6 months later for comparison. We assessed these changes with a 2-tailed Mann-Whitney U test with significance set at P<.05.

Postintervention conference presentation at Site A increased 302% compared to its previous 3-year average (n=34, P<.001), and 91% compared to its preintervention year (n=34, P<.001; Tables 1 and 2). Residents at site A had no significant change to reported textbook chapters or research participation. Residents at site B had no significant change in any scholarly activity reported to the ACGME.

Site A residents perceived a 217% increase in case submissions (n=12, P=.0198) after the CRW, but no other significant increases in how they viewed cases (Table 3). Site B residents perceived a 217% increase in case consents (n=12, P=.0404), a 163% increase in case reports drafted (n=12, P=.0316), and a 1,000% increase in cases submitted (n=12, P=.0383), but no other significant increases in how they viewed cases.

External constraints prevented our planned statistical analysis of resident perceptions. We intended to use a Wilcoxon Signed Rank test. However, because of technical constraints, we were unable to pair pre and postsurveys. That technical limitation also means that we cannot say for certain that all respondents to the postsurvey in fact participated in the CRW and completed a presurvey; for this reason our survey data does not meet strict criteria for any statistical test. However, it is unlikely that any resident who did not participate in the CRW and complete the presurvey would have completed a postsurvey. We therefore report the survey data using a Mann-Whitney U test, with the caveat that technical criteria for using that test are not met.

This study shows a correlation between the curriculum intervention and increased scholarly output at the residency site implementing all three components. Further study is needed to identify a causal link to this observed correlation, address the limitations of this study, and confirm that the correlation observed was not caused by the baseline differences between the programs.

Those baseline differences in residency characteristics and lack of randomization preclude claiming a correlation between the RRC position and an increase in scholarly activity. However, it is noteworthy that the program that adopted the RRC had significant objective improvement in resident productivity, while the site without an RRC improved only in resident perceptions. Perhaps filling knowledge gaps with interventions like the CRW and The Recipe guidebook are necessary but insufficient to increase resident scholarship, and resident peer leaders allow residents to overcome the uncertainty barriers to scholarly activity described by Ledford et al.6 A qualitative study may help identify a correlation, and would also help identify other characteristics of the culture of scholarship that appears to be a common theme across successful scholarly activity interventions.8 Other limitations to our study include short follow-up time, which may explain why conference participation increased, but longer lead time outcomes like textbook chapters and research participation did not.

Because of the technical limitations in secondary outcomes described above, the survey data must be considered anecdotal. The surveys were deidentified, so it is not possible to directly compare resident self-assessment with actual output. Further limitations include small sample size and implementation only in larger military residencies. However, as none of these interventions are military-specific, and each program operates under the same ACGME and American Board of Family Medicine requirements, we think it likely that the interventions are generalizable to civilian programs.

Another limitation is the larger size of these programs; smaller programs may find it more challenging to identify and support a resident peer leader. However, a strength of the study is the low cost of these interventions in time and money, and hence their utility even to programs with limited resources. The CRW requires a 3-hour time commitment, and has materials for presentation freely available. The Recipe guidebook is also freely available. Resident peer leaders require no direct compensation. NHJ has given an average of 2 weeks of time over the course of a year to its resident peer leaders. This is usually distributed as one half day per month, with extra time around submission and conference deadlines, and is counted as a research elective.

Another strength of this study is its use of ACGME-reported scholarly activity to measure improvements. This overcomes prior study limitations using “quality” or “high quality” measures that lacked universal use, and helps to demonstrate the generalizability of our results. Another strength of this study is its prospective comparison of similar programs.

The correlation between the curriculum and scholarly output at two additional programs is promising. Future opportunities for research include longevity evaluations of site A’s successes and larger randomized controlled trials including smaller programs and civilian programs to address the limitations outlined above. Including a qualitative component to these studies may also help determine causative elements of changes to programs’ research outputs.