Outcomes of a Health Systems Advocacy, Leadership, and Management Curriculum in a Family Medicine Residency Program

Only a small proportion of US hospitals are physician-led.¹ Physicians often are identified and promoted to leadership positions based on career achievements rather than management and leadership skills.^2,3 Some medical schools and residency programs have shown innovation in this space.^4,5 A major step in the right direction is the establishment of a review committee specific to Healthcare Administration, Leadership, and Management (HALM) by the Accreditation Council for Graduate Medical Education. The reported purpose of HALM is to provide physicians with the education, mentorship, and expertise necessary to positively contribute to various aspects of patient care, including patient safety, health care quality, care management, and systems of care.⁶ HALM fellowship programs currently are active and accredited at the Cleveland Clinic Foundation in Ohio and the Icahn School of Medicine at Mount Sinai in New York City. These leadership training programs provide physicians with the skills necessary to become physician leaders not by accident, but based on competency.⁴

Some of the core HALM competencies are already common program requirements.⁷ Although most residency programs and some medical schools offer a curriculum in health systems, a dearth of data exists on the outcomes of these curricula from students, residents, and fellows. To facilitate resident competency, the HALM training and education curriculum could easily be adopted by programs locally.^8-10 We implemented a HALM curriculum in a family medicine residency program featuring competencies in patient safety, health care quality, care management, and systems of care. This study is a retrospective review of Kirkpatrick Level 2, 3, and 4 outcomes of the 5 years of implementation of the HALM curriculum.

We conducted a quasi-experimental study with pre- and postdesigns to assess the outcomes of the HALM curriculum using Kirkpatrick Level 2, 3, and 4 behaviors. Kirkpatrick levels represent a model used for measuring the outcomes of a curriculum.^11-13 Our study specifically defined each Kirkpatrick level as shown in Table 1. Kirkpatrick Level 1, a reaction to the curriculum, was omitted from this study because all residents were able to achieve Level 1. The majority of Level 2 Kirkpatrick outcomes, leadership within the program, were elected and designated positions by the program. These were then entered into the New Innovations (QGenda) residency management suite to hold the data. Level 3 Kirkpatrick outcomes, leadership outside the program, also were verified by the program coordinator once input into New Innovations. All leadership in quality improvement and patient safety projects were confirmed by the attending associated with the team. Level 4 outcomes, which consisted of leadership roles postgraduation, were abstracted from the data collected and maintained by an alumni engagement team. These data were collected by the investigators of this study from surveys already conducted by the program as part of graduate engagement from a separate project related to resident recruitment and retention postgraduation. The information routinely collected included several data points. This study used these data points: name, current institution, role/designation, scope of practice. This study used that database to discern the Kirkpatrick Level 4 outcomes, leadership role postgraduation, from role/designation and scope of practice. The resident recruitment and retention postgraduation team collected data from the graduates with annual surveys sent via multiple modes, including email, paper mail, phone call, text, and social media (encompassing existing WhatsApp class groups). Their methodology involved continual engagement of the graduates using different modes of communication until the team was able to update the information in their database. Kirkpatrick levels were collected as count data for each resident, with responses consisting of either yes or no.

Using a retrospective approach, we assessed the outcomes following an innovative 3-year longitudinal curriculum on HALM for family medicine residents. The curriculum was implemented at the start of residency and continued until graduation. This mandatory curriculum was designed using concepts from Kern’s model, as shown in Figure 1. A capstone project, geared toward leadership and interprofessional improvement, counted toward scholarly activity. As previously reported by Waheed and colleagues, a modified Seehusen score was used for all scholarly activity requirements.¹⁴ Table 2 shows key highlights of the curriculum.¹⁵ The curriculum provided residents with systems thinking and experience in measuring and improving quality of care, patient safety, health care access, advocacy and organized medicine, practice management with population health, and addressing social determinants of health.

The outcomes measured included meaningful resident engagement and involvement in a large, complex, integrated health care delivery system, as shown in Table 1. This study was determined to be exempt from review by the WellSpan Health Institutional Review Board via letter number 1887432-1.

This study evaluated two cohorts of residents. The preintervention group, before implementation of HALM, included the residents who graduated between 2016 and 2020. The postintervention group, after implementation of HALM, included the residents who graduated between 2021 and 2024. We measured the impact of HALM by assessing whether residents met one or more of the Kirkpatrick Level 2 (leadership within the program), 3 (leadership outside the program), and 4 (leadership role postgraduation) outcomes before and after implementing the HALM curriculum. The data on resident demographics and achievement of Kirkpatrick level criteria pre- and postimplementation were collected as count data through work submitted into the residency management software. We used the χ² and Fisher exact tests to compare the characteristics of pre- and postgroups. We used univariate regression analysis to detect significant differences in the outcomes between pre- and postgroups using SPSS 29 (IBM).

A total of 51 residents, 26 in pre- and 25 in postgroup, were included in this study. Table 3 shows the demographics of each group, including gender, race, and type of medical school (Commission on Osteopathic College Accreditation accredited, Liaison Committee on Medical Education (LCME) accredited, or international). Gender was categorized into female vs male, and race was measured among White, Asian, Black, and Latinx participants. The pre- and postimplementation groups were comparable with no statistically significant difference in their composition (P >.05) regarding gender and race. The number of residents from Osteopathic schools was significantly higher in the pre-implementation group than in the post (P<.001).

For Level 4 (leadership role postgraduation) outcomes, this study was unable to calculate exact response rates of the surveys. This is because the data used were previously captured from the graduate surveys as described in the Methods section. The near 100% response rate for these annual surveys was due to continual follow-up until a response was received. Multiple modes of communication, including email, WhatsApp, and paper mail, were used. The high response rate was likely due to 80% of graduates working in communities that the sponsoring institution (WellSpan Health) serves. The 40% to 60% retention rate at WellSpan plus another 20% to 40% with other groups part of the WellSpan Accountable Care Organization have provided us with ready access and continual connection with residency graduates.

All residents in the post-HALM implementation group demonstrated leadership of a project within the program, while more than two-thirds also led a project with scope outside of the program/institution. Table 4 shows the results of univariate logistic regression pre- and postimplementation of the HALM curriculum. The odds of significant leadership roles and competencies were higher in the postimplementation group except for the chief resident designation, which is a fixed number allowed by the institution. Interesting to note is that 32% of the graduates from the postimplementation group already have obtained a significant physician leadership role after graduation—a number that may continue to grow within the outlined 5-year timeline for the post-HALM group.

The implementation of the HALM curriculum was associated with improved resident competency in health systems science related to health care advocacy, leadership, and management, as measured by Kirkpatrick outcome classifications. This innovative curriculum can serve as a model for other residency and fellowship programs seeking to fulfill accreditation requirements and equip future physicians with essential skills.

The odds ratios between outcomes pre- and postimplementation indicate the positive association of this HALM curriculum with residents’ competency achievement. The effect was seen at all levels, including achieving significant leadership within the program (Kirkpatrick Level 2), the organization/institution, and/or nationally during residency training (Kirkpatrick Level 3). The Kirkpatrick Level 4 behavior (leadership role postgraduation) associated with implementation of the curriculum was particularly interesting. In the 5 years before the implementation of the HALM curriculum, only two learners had achieved this level; postimplementation, that number increased to eight learners. The 2019 National Graduate survey showed that fewer than 10% achieved a significant leadership role,¹⁶ while 32% of the graduates exposed to our HALM curriculum achieved one. Cook and colleagues also reported only a modest increase in their outcomes in Kirkpatrick Level 2 (leadership within the program) and beyond.¹⁷

Limitations of this study included a small sample size, information collected from only one residency program of a smaller size, the length of time between pre- and postimplementation groups for data collection, virtual vs in-person learning due to the COVID-19 pandemic, and the type of medical school of the graduates as confounder. We found no statistical difference between pre- and postimplementation cohorts with regard to gender and race. However, our study included considerably more osteopathic graduates than international and LCME accredited students in the preimplementation group compared to the postimplementation group. Whether this factor would have affected the differences seen in this study is unclear. Furthermore, if this curriculum were to be implemented in a program with a larger number of residents per year or longer accreditation length of the program, whether the results would be replicable is unclear because having smaller cohorts advantageously allows for closer interaction with faculty and opportunity for application of curriculum. For example, internal medicine programs often have cohorts double or triple in size, and other specialty programs such as obstetrics and gynecology or general surgery have a greater number of years as the accredited length of the programs. This study also relied on self-reported designation and role of graduates for the Level 4 (leadership role postgraduation) outcomes, which could be seen as a limitation. However, we could imagine no apparent reason for graduates to report a designation, title, or role such as medical director if they were not because no benefit is gained from doing so.

Physicians often are seen as leaders in health care. However, like every other skill, leadership must be learned.^18-21 This HALM curriculum’s outcomes hold promise for enhancing leadership in health systems. The next step is to expand the HALM curriculum to other residency programs and to collect data on their outcomes.