Use of Epic Signal Data to Evaluate the Impact of an “EHR Pearls” Curriculum

Medical residents spend significant time learning and using electronic health records (EHR).¹ While multiple studies of providers have demonstrated associations with EHR, stress, and burnout,^2-4 EHR training programs can have a positive impact on EHR proficiency and burnout.^5-8 In 2020 we developed an EHR training program (the “EHR Pearls curriculum”) with the goal of reducing stress through improved efficiency. To assess the impact of this training on EHR usage, we analyzed Epic Signal data (Epic Systems Corporation, Inc.) of performance metrics across multiple training years and specialties. These vendor-developed indices include a Provider Efficiency Profile Score (“efficiency”), a comparative value (1-10) for time spent in the EHR relative to the provider’s estimated workload. Additionally, the Proficiency Score (“proficiency”) is a comparative value (0-10) based on use of specific Epic tools. While studies have shown proficiency improvements through focused interventions, ^9-10 no studies have looked at the impact of training in a residency program. The objective of this study was to assess the value of our training in impacting the Epic Signal value surrogates for proficiency and efficiency.

Starting in 2020, family medicine (FM) residents in this 10-resident-per-year, urban academic program in Southwest Virginia completed 10 hours of focused training during orientation. The training was built around customization and optimization of the EHR’s clinical documentation features. This instruction was augmented through a series of quarterly lectures on relevant topics. Additionally, second-year residents received semiannual one-on-one instruction with a health system EHR trainer.

For this retrospective cohort study, we extracted 3 years of deidentified, monthly Signal data for all residents in the FM Residency as well as for comparator residents at the same institution who were not exposed to focused EHR residency curricula (internal medicine [IM], obstetrics and gynecology [OB-GYN], and pediatrics). The analysis focused on the 7-month periods of September 2019 through March 2020 (preintervention) and September 2020 through March 2021 (postintervention) for the FM residency as well as September 2021 through March 2022 when comparing between residency programs. We chose the initial 6-month period to avoid confounding elements including late graduates of the program (before September) and initial impacts of COVID on the workflow (after March). We excluded resident data when it was incomplete (due to leaves of absence and unusual start times). A total of 18 FM residents and 26 other residents were included in the analysis (Table 1). 

We monitored trends in efficiency and proficiency longitudinally between class years and all four residency programs. We averaged each department’s individual resident scores together to provide the resident score per department per month, for both the efficiency and proficiency. We compared the trends between the postgraduate year (PGY) 1 residents vs PGY-2 residents within each department over the academic years. Based on the observed differences in those trends, specifically between FM and IM departments, we conducted a more in-depth analysis comparing their residents. Since it was expected that a resident’s efficiency and proficiency would change with experience over the course of the academic year, we used an analysis of covariance to compare the scores between resident levels within each department while controlling for that change over time, as well as scores between the PGY-2s within the FM and IM departments. This study was reviewed and exempted from oversight by the Carilion Clinic Institutional Review Board for the Protection of Human Subjects.

Proficiency data demonstrated consistent improvement in performance for FM PGY2s vs PGY-1s. This was true for both the 2020-2021 (P<.0001) and 2021-2022 (P<.0001) academic years (AY; Figure 1). This was not the case for the comparator residency programs (IM: P=.0653; Peds: P=.1160; OB-GYN: P=.4540; data available by request). In the IM-specific analysis, the proficiency for the FM PGY-2 residents in 2022 was significantly higher than the IM PGY-2 scores (P<.0001) when controlling for time in the program that year (Figure 2).

In contrast, the efficiency data demonstrated a consistent decrease in efficiency for FM PGY2s compared to FM PGY1s. This was true for both the 2020-2021 (P<.0001) and 2021-2022 (P<.0001) academic years (Figure 3). The comparator residency programs’ timelines (IM: P=.4280; Peds: P=.1790; OB-GYN: P=.4890) did not demonstrate any consistent change in efficiency from first to second year of training (data available upon request). Further, the efficiency for the FM PGY-2 residents in 2022 was significantly lower when compared specifically to the IM PGY-2 scores (P<0.0001) when also controlling for time in the program that year (Figure 4). Median efficiency for the FM cohorts dropped from 5 during PGY-1 to 4 during PGY-2. In contrast, the median score remained at 5 for IM cohorts and FM faculty.

While Epic Signal data have been available for analysis of provider behavior for nearly a decade, we could not find reports of its use in evaluating training curricula in a residency. This study demonstrated the potential utility of Signal data in evaluating a training program for EHR clinical documentation.

Improvements in FM resident proficiency over their first 2 years were consistent with expected gains from the “EHR Pearls” curriculum While the magnitude of that improvement relative to the expected progression from experiential learning could not be clearly distinguished, the lack of similar improvement in the comparative programs suggested the training was impactful.

Conversely, FM residents were less efficient during their second year as measured in their time spent in the EHR relative to their clinic volume. Since FM faculty and PGY-1 residents had median efficiency scores of 5, the significant drop for PGY2s to a median efficiency of 4 was unexpected. This finding could have been related to factors for PGY2s such as a doubling in panel size and change in workload complexity with introduction of acute care visits and a reduction in panel continuity. Khirat et al in 2023 suggested an inverse correlation between proficiency and efficiency in pediatric specialists, potentially due to increased complexity of care.¹¹ Gaining proficiency with the EHR tools may have paradoxically resulted in more comfort with the platform with longer documentation time rather than cursory notes. Additionally, gender disparities in EHR use may have played a role as women physicians have been shown to have higher proficiency¹² while also spending more time in the EHR compared to male physicians.^13-16 Further exploration of this discrepancy could include Signal elements such as length of notes to see if other targets of efficiency could be addressed individually or as a group.

Limitations of the analysis related to small sample size of residents per year (10 in FM) and variability in individual engagement and competence with the EHR. Monthly rotation differences (inpatient vs outpatient focused) may have resulted in variability in the efficiency scores. The vendor-defined variables also do not reflect the full breadth of tasks required by residents when interacting with the EHR.

While limits to the Signal data exist, the study did show the training program that we created had a positive impact on proficiency. These results are expected to be generalizable to other institutions regardless of the EHR platform in use. Additional steps include looking at tools to improve efficiency (such as greater emphasis on voice recognition software) and repeating the analysis after those interventions are integrated.