Introduction: Primary care is evolving to meet greater demands for the inclusion of collaborative health care quality improvement (QI) processes at the practice level. Yet, data on organizational preparedness for change are limited. We assessed the feasibility of incorporating an organizational-level readiness-to-change tool that identifies factors relevant to QI implementation at the practice level impacting new family medicine physicians.
Methods: We assessed organizational readiness to change at the practice level among residents participating in a team-based QI training curriculum from April 2016 to April 2019. Seventy-six current and former residents annually completed the modified Organizational Readiness to Change Assessment (ORCA) survey. We evaluated QI and leadership readiness among five subscales: empowerment, management, QI, QI leadership (skills), and QI leadership (ability). We calculated mean survey scores and compared across all 3 years. Resident interviews captured unique perspectives and experiences with team-based activities. Qualitative analysis identified emergent themes.
Results: Residents completed 73 modified ORCA surveys (96% response rate). Compared to years 2016-2019, 2018 results were highest in mean negative responses for the QI subscale (24.62, SD 6.70). Four volunteers completed postsurvey interviews. Qualitative analysis identified issues concerning communication, team collaboration, practice site functioning, and survey relevance.
Conclusions: Our study determined that miscommunication and practice site disruptions undermine organizational-level readiness to change, as measured by the ORCA tool which was part of a multimethod assessment included within a team-based QI training curriculum. Training programs undergoing curricula transformations may feasibly incorporate ORCA as a tool to identify impediments to collaborative practice and inform resource allocation important for enhancing physician training in QI leadership.
Primary care is evolving to meet increasing demands for clinical practice transformation resources that sustain population health.1-3 Today, new family physicians face distinct challenges as practice quality improvement (QI) leaders. Residency curricula now include team-based experiential learning to equip clinicians with the skills needed to revamp quality of care across health care organizations.4 Yet, data on the organizational preparedness of residency programs to incorporate such changes are limited.
The Organizational Readiness to Change Assessment (ORCA) survey is a validated instrument with proven scale reliability in measuring organizational-level preparedness for effecting evidence-based changes in clinical practice.5 The ORCA survey was developed by the Veterans Health Administration in accordance with core components of the Promoting Action on Research Implementation in Health Services theoretical framework.5 This model is commonly used for QI implementation assessment within healthcare settings.
In this study, we assessed organizational readiness to change at the practice level among family medicine residents. Utilizing a multimethod approach, we collected data from residents who participated in team-based QI and leadership training at a large, urban practice site in Philadelphia, Pennsylvania. Our goal was to determine the feasibility and utility of incorporating an annual ORCA tool within our family medicine residency curriculum.
In 2015, Thomas Jefferson University’s family medicine residency program enhanced its QI and leadership curriculum as part of a Health Resources Services Administration Primary Care Training and Enhancement grant. We used an online QI toolkit of learning modules from the Institute for Healthcare Improvement as the framework for development.4 Weekly conferences included dedicated time for resident participation in team-based QI projects. Teams developed aim statements and created plan-do-study-act cycles.4 Project results were presented at grand rounds and an annual institutional QI poster session. A modified ORCA survey was introduced as an interventional tool to annually assess residents’ perspectives of practice level readiness-to-change.5
Quantitative and Qualitative Measures
Each spring, residents completed a modified ORCA composed of five subscales measuring QI and leadership readiness: empowerment (action), management (facilitation), QI (problem solving), QI leadership (skills), and QI leadership (ability). Residents read a series of statements associated with each subscale, such as, “I have the skills, knowledge and ability to promote team building to solve clinical care problems.” A 5-point Likert scale captured levels of agreement or disagreement (strongly disagree to strongly agree). Scores were calculated as means. High or low mean scores reflect positive or negative agreement.
Of the eligible 76 current and former family medicine residents who received training between 2016 and 2019, 73 completed ORCA surveys. Noting increased negative mean results among 2018 residency graduates, faculty members determined follow-up interviews may yield more in-depth data. Negative responses determined to be associated with specific survey subscales (QI leadership and management) informed the development of four open-ended interview questions by the faculty and research team, an inquiry approach based on modified grounded theory.6
A single research team reviewer conducted separate interviews with four residents who volunteered to meet at preselected locations of their choosing. The residents’ identities were withheld from faculty. Each participant was asked four open-ended questions pertaining to survey relevancy and practice change barriers such as, “Do you think the survey questions are appropriate for capturing information concerning the activities of the QI team projects?” and “Which areas or activities do you think were especially challenging for residents and why?” The single reviewer documented all responses by hand and completed all processes of qualitative analysis.
We calculated descriptive statistics for all survey scores in Microsoft Excel. Means and standard deviations were compared across scores from 2016-2019. Mean differences between 2018 residency graduates and 2019 PGY-3 residents were examined to identify trends. We used an open-coding approach to assess open-ended responses. We performed qualitative analysis with NVivo software to identify emergent themes. The Institutional Review Board of Thomas Jefferson University determined this study to be exempt from review.
With a response rate of 96% (73/76), data analysis results determined ORCA scores trended negatively from 2016-2018. A marked increase in disagree or strongly disagree responses occurred in 2018. Moreover, negative responses were specific to subscale, question type, and resident year. For example, the QI subscale had the lowest mean disagree responses for 2018 PGY-3 learners (23.86, SD 7.88) when compared to 2016 and 2017 PGY-3 learners, respectively (29.79, SD 2.11 and 27.29, SD 3.35; Table 1). Also, strongly disagree responses were selected most often for statements concerning empowerment, influence, communication, goal determination, and role assignment.
Qualitative analysis identified three emergent themes specific to adverse events experienced by residents (Table 2). Themes centered on miscommunication, team collaboration, and practice site functioning. High turnover in support staff was also found to hinder QI project execution. Residents further reported that ORCA survey jargon was discordant with day-to-day practice activities and the curriculum, as evidenced by the statement, “For example, the word ‘deliverables’ is foreign to our world.” No negative comments about the program, faculty, clinical staff, or other residents were observed.
Postgraduate Year Trends
Data pattern comparisons revealed trending scores differed between 2016-2018 and 2017-2019 residents. In contrast to 2018 graduates, mean scores trended positively for 2019 residents. Curriculum and survey clarity improvements may have contributed to increases in mean positive responses (Figures 1-2).
Multimethod data analysis determined that obstacles to team-based practice and organizational readiness-to-change include miscommunication, misaligned interpersonal group dynamics, and practice site dysfunctions. ORCA results for 2018 residents revealed that negative response patterns were specific to survey subscales, in alignment with qualitative findings. However, 2019 resident data results suggest curriculum improvements had a positive effect on residents’ QI and leadership training experiences.
Our study demonstrates that the modified ORCA is feasible and useful to administer to family medicine residents. When coupled with qualitative interviews, actionable feedback facilitates organizational readiness-to-change activities. The implications of these findings are particularly relevant to primary care training programs. ORCA is a valuable tool for guiding health care organizations with evidence-based clinical QI practice implementation, early identification of potential issues, and annual assessment of learner progress.
The authors thank Dr Michael Rosenthal and Rashida Smith, MPH, for their inspiration, support, and generous feedback.
Financial support: This work was funded by the Health Resources and Services Administration of the US Department of Health and Human Services Grant Number: TOBHP28581. Project title: “Accelerating Primary Care Transformation at Jefferson (JeffAPCT).”
Presentations: This study was presented at the Society of Teachers of Family Medicine Annual Spring Conference, April 27-May 1, 2019, Toronto, Canada.
- American Academy of Family Physicians. Integration of Primary Care and Public Health (Position Paper). https://www.aafp.org/about/policies/all/integprimarycareandpublichealth.html. Accessed October 11, 2019.
- Marlowe DP, Manusov EG, Teasley DJ. A team-building model for team-based care. Fam Pract Manag. 2012;19(6):19-22. https://www.aafp.org/fpm/2012/1100/p19.html. Accessed October 11, 2019.
- Anderson DR, Olayiwola JN. Community health centers and the patient-centered medical home: challenges and opportunities to reduce health care disparities in America. J Health Care Poor Underserved. 2012;23(3):949-957. doi:10.1353/hpu.2012.0099
- Institute for Healthcare Improvement. Quality Improvement Essentials Toolkit. http://www.ihi.org/resources/Pages/Tools/Quality-Improvement-Essentials-Toolkit.aspx. Accessed October 8, 2019.
- Helfrich CD, Li Y-F, Sharp ND, Sales AE. Organizational readiness to change assessment (ORCA): development of an instrument based on the Promoting Action on Research in Health Services (PARIHS) framework. Implement Sci. 2009;4(38):38. doi:10.1186/1748-5908-4-38
- Chun Tie Y, Birks M, Francis K. Grounded theory research: A design framework for novice researchers. SAGE Open Med. 2019;7:2050312118822927. doi:10.1177/2050312118822927