Organizational Readiness to Change: Quality Improvement in Family Medicine Residency

Primary care is evolving to meet increasing demands for clinical practice transformation resources that sustain population health.^1-3 Today, new family physicians face distinct challenges as practice quality improvement (QI) leaders. Residency curricula now include team-based experiential learning to equip clinicians with the skills needed to revamp quality of care across health care organizations.⁴ Yet, data on the organizational preparedness of residency programs to incorporate such changes are limited.

The Organizational Readiness to Change Assessment (ORCA) survey is a validated instrument with proven scale reliability in measuring organizational-level preparedness for effecting evidence-based changes in clinical practice.⁵ The ORCA survey was developed by the Veterans Health Administration in accordance with core components of the Promoting Action on Research Implementation in Health Services theoretical framework.⁵ This model is commonly used for QI implementation assessment within healthcare settings.

In this study, we assessed organizational readiness to change at the practice level among family medicine residents. Utilizing a multimethod approach, we collected data from residents who participated in team-based QI and leadership training at a large, urban practice site in Philadelphia, Pennsylvania. Our goal was to determine the feasibility and utility of incorporating an annual ORCA tool within our family medicine residency curriculum.

In 2015, Thomas Jefferson University’s family medicine residency program enhanced its QI and leadership curriculum as part of a Health Resources Services Administration Primary Care Training and Enhancement grant. We used an online QI toolkit of learning modules from the Institute for Healthcare Improvement as the framework for development.⁴ Weekly conferences included dedicated time for resident participation in team-based QI projects. Teams developed aim statements and created plan-do-study-act cycles.⁴ Project results were presented at grand rounds and an annual institutional QI poster session. A modified ORCA survey was introduced as an interventional tool to annually assess residents’ perspectives of practice level readiness-to-change.⁵

Each spring, residents completed a modified ORCA composed of five subscales measuring QI and leadership readiness: empowerment (action), management (facilitation), QI (problem solving), QI leadership (skills), and QI leadership (ability). Residents read a series of statements associated with each subscale, such as, “I have the skills, knowledge and ability to promote team building to solve clinical care problems.” A 5-point Likert scale captured levels of agreement or disagreement (strongly disagree to strongly agree). Scores were calculated as means. High or low mean scores reflect positive or negative agreement.  

Of the eligible 76 current and former family medicine residents who received training between 2016 and 2019, 73 completed ORCA surveys. Noting increased negative mean results among 2018 residency graduates, faculty members determined follow-up interviews may yield more in-depth data. Negative responses determined to be associated with specific survey subscales (QI leadership and management) informed the development of four open-ended interview questions by the faculty and research team, an inquiry approach based on modified grounded theory.⁶

A single research team reviewer conducted separate interviews with four residents who volunteered to meet at preselected locations of their choosing. The residents’ identities were withheld from faculty. Each participant was asked four open-ended questions pertaining to survey relevancy and practice change barriers such as, “Do you think the survey questions are appropriate for capturing information concerning the activities of the QI team projects?” and “Which areas or activities do you think were especially challenging for residents and why?” The single reviewer documented all responses by hand and completed all processes of qualitative analysis.

We calculated descriptive statistics for all survey scores in Microsoft Excel. Means and standard deviations were compared across scores from 2016-2019. Mean differences between 2018 residency graduates and 2019 PGY-3 residents were examined to identify trends. We used an open-coding approach to assess open-ended responses. We performed qualitative analysis with NVivo software to identify emergent themes. The Institutional Review Board of Thomas Jefferson University determined this study to be exempt from review.

With a response rate of 96% (73/76), data analysis results determined ORCA scores trended negatively from 2016-2018. A marked increase in disagree or strongly disagree responses occurred in 2018. Moreover, negative responses were specific to subscale, question type, and resident year. For example, the QI subscale had the lowest mean disagree responses for 2018 PGY-3 learners (23.86, SD 7.88) when compared to 2016 and 2017 PGY-3 learners, respectively (29.79, SD 2.11 and 27.29, SD 3.35; Table 1). Also, strongly disagree responses were selected most often for statements concerning empowerment, influence, communication, goal determination, and role assignment.

Qualitative analysis identified three emergent themes specific to adverse events experienced by residents (Table 2). Themes centered on miscommunication, team collaboration, and practice site functioning. High turnover in support staff was also found to hinder QI project execution. Residents further reported that ORCA survey jargon was discordant with day-to-day practice activities and the curriculum, as evidenced by the statement, “For example, the word ‘deliverables’ is foreign to our world.” No negative comments about the program, faculty, clinical staff, or other residents were observed.

Data pattern comparisons revealed trending scores differed between 2016-2018 and 2017-2019 residents. In contrast to 2018 graduates, mean scores trended positively for 2019 residents. Curriculum and survey clarity improvements may have contributed to increases in mean positive responses (Figures 1-2).

Multimethod data analysis determined that obstacles to team-based practice and organizational readiness-to-change include miscommunication, misaligned interpersonal group dynamics, and practice site dysfunctions. ORCA results for 2018 residents revealed that negative response patterns were specific to survey subscales, in alignment with qualitative findings. However, 2019 resident data results suggest curriculum improvements had a positive effect on residents’ QI and leadership training experiences.

Our study demonstrates that the modified ORCA is feasible and useful to administer to family medicine residents. When coupled with qualitative interviews, actionable feedback facilitates organizational readiness-to-change activities. The implications of these findings are particularly relevant to primary care training programs. ORCA is a valuable tool for guiding health care organizations with evidence-based clinical QI practice implementation, early identification of potential issues, and annual assessment of learner progress.