Introduction: The Accreditation Council for Graduate Medical Education (ACGME) requires family medicine residents to complete a quality improvement (QI) project. There is a need for more QI training activities to be shared to meet this requirement. Our objective was to describe an activity for residents to improve women’s preventive health services in an underserved clinic. Specific aims were to determine: (1) how women’s receipt of preventive services compared to benchmarks, (2) physician and staff knowledge of the process and barriers to receiving services, and (3) whether an intervention to increase awareness among physicians and staff improved preventive services.
Methods: Residents (N=30) evaluated charts (N=505) to determine receipt of mammograms, pap tests, colon cancer screenings, and pneumonia vaccines. We compared estimates to existing clinic benchmarks. We presented initial (preintervention) results to physicians and staff at clinic team meetings. We collected perceptions of processes and barriers to preventive services. Preintervention methods were replicated (N=100) and results were compared (postintervention).
Results: Preintervention, mammograms (72%) and Pap tests (65%) were lower than clinic benchmarks. Most (81%) women ages 65 and older received a pneumonia vaccine; however, this was lower than the national Healthy People 2020 goal. Fear, knowledge, and scheduling were identified as top barriers. Post-intervention, there was a statistically significant increase in Pap tests (P=.0013).
Conclusion: This activity trained residents how to impact their practice through QI methods and can be used in other programs as a foundation for developing basic QI initiatives. Future efforts should focus on evaluating barriers to preventive services from the patient perspective.
To meet Accreditation Council for Graduate Medical Education (ACGME) requirements, residents are required to complete a quality improvement (QI) project.1 Impactful QI residency training programs involve clinical outcomes and metrics, direct role modeling, and evaluations of patient impact.2,3 Most QI curricula focus on learner knowledge.2-4 Primary care-based QI education models should incorporate experiential learning and QI application.4-8 More QI curricula that focus on evaluation of clinical outcomes while educating learners are needed.9,10
We developed a QI project to improve women’s preventive health services using experiential learning activities for FM residents who practice in an underserved clinic. We conducted a quasi-experimental project to determine: (1) how women’s receipt of vaccines and cancer screenings compared to benchmarks, (2) physician and staff knowledge of processes and barriers to receiving preventive services, and (3) whether an intervention for physicians and staff improved patients’ receipt of preventive services.
The project was developed for the Parkland Family Medicine Clinic, the training site for family medicine residents. The clinic conducts approximately 20,000 visits per year among medically underserved residents in Dallas, Texas. More than 60% of patients have limited or no English proficiency.
Preintervention, we randomly selected electronic medical records (EMRs, N=505) from 2,500 female patient (ages 40 years and older) visits during a 4-month period, based on a clinic report. First-year residents reviewed 10 charts while second- and third-year residents reviewed 20 charts. We evaluated charts for: (1) mammogram in past 2 years, (2) Pap test in past 3 years, (3) colon cancer screenings (ages 50 years or more), and (4) pneumonia vaccine (ages 65 years or more) based on recommendations.10-12 We compared estimates to local (Dallas County),13 state (Texas),14 and Healthy People 2020 estimates/goals.15
During the intervention, preliminary results and an overview of recommendations were presented to physicians and staff. We collected surveys assessing knowledge and. Two questions were asked for each preventive service. Participants were asked, “what do you know about the process for a patient at our clinic to receive [insert preventive service] after a provider has ordered it?” Then, participants were asked “what do you think are some potential barriers to patients at our clinic getting a mammogram/Pap test/colonoscopy/pneumonia vaccine?”
Postintervention, a new random sample of records (n=100) were selected from patient visits the following year. Our postintervention sample size was smaller because data could only be collected by three residents, who used this project to meet research and scholarly activity requirements, instead of by all residents. We compared pre- and post results.
We used descriptive statistics to report age and receipt of preventive services. We developed themes to describe processes/barriers. We entered open-ended responses into a Microsoft Excel database, and identified key words related to processes and barriers, and we counted recurring words (ie, scheduling, language). We used McNemar tests to determine statistically significant differences pre-/postintervention. This project was reviewed and approved as QI by our institutional review board.
Preintervention, most women ages 40 years and older received mammograms (72%) and Pap tests (65%). Findings were lower than Healthy People 2020 goals (mammograms=81%; Pap tests=93%) and local (mammograms=75%; Pap tests=77%) estimates. Clinic estimates for mammograms (72%) were higher than statewide estimates (66%). Clinic estimates for Pap tests (65%) were lower than statewide estimates (76%). Among women ages >50 years, over half (56%) received a colon cancer screening, with 53% receiving up-to-date screening. Most (81%) women ages 65 years or more received a pneumonia vaccine, which was lower than the Healthy People 2020 goal (90%, Table 1).15
We found several inconsistencies when we measured processes for receiving services (Table 2). For mammograms, 34% reported that patients had the responsibility to set up the appointment, while 15% reported that patients were called for appointments. Similar inconsistencies were found for colonoscopy. For Pap testing, 41% reported that patients had to set up separate appointments, whereas 20% indicated that they could be done during already-scheduled appointments. Health care providers and staff reported communication barriers with patients about pneumonia vaccines. The most commonly reported processes were physicians ordering the vaccine (32%) and discussing the vaccine with patients (11%). Using descriptive data, fear and lack of knowledge were identified as top barriers. Scheduling difficulties were identified as barriers to completing mammograms (42%), colonoscopies (50%), and Pap tests (15%). Limited stock was a barrier for pneumonia vaccines (19%).
Despite not reaching statistical significance (P<.05), we found improvements for colon cancer screenings and pneumonia vaccines postintervention (Table 1). Mammograms decreased only slightly. Pneumonia vaccines met Healthy People 2020 targets (90%). We found a statistically significant increase (P=.0013) in the proportion of women who received Pap tests despite not meeting Healthy People 2020’s goal. Up-to-date colon cancer screenings (70.0%) were higher than Texas estimates (62.6%) and similar to Healthy People 2020 (70.5%) goals.
Our goal was to develop an activity for family medicine residents to learn how to conduct QI projects while improving women’s preventive health services. In accordance with recent QI practice, another focus was to shed light on clinic performance in providing women’s preventive services. We observed the following results.
Prior to completing this project, our clinic had poor performance on Pap tests and colon cancer screenings. Postintervention, our clinic met Healthy People 2020’s goal for pneumonia vaccines (90%) among women aged 65 years and older. Furthermore, we observed a significant increase in Pap testing (20.4% increase, Table 1) after intervention. Our results are similar to previous studies that found that Pap testing improved after the development of a women’s preventive health clinic for residents.16,17 We observed a slight decrease in mammogram uptake (preintervention 72.2%; postintervention 69.1%). In addition to the smaller sample size, one possible explanation may be a difference the patients’ age. Preintervention, 20% of the sample was aged 40-49 years, versus 30% of the sample postintervention.
During our intervention with physicians and staff, several inconsistencies in the process for obtaining screenings and vaccines were reported. There was lack of agreement on the process of setting up appointments. This inconsistency may have led to scheduling difficulties being reported as a top barrier for all screenings. Patient-provider communication and system-level coordination have been identified as strategies for developing best practices and streamlining processes.18,19
Strengths and Limitations
Strengths of this study include a large sample of patients’ records available to review and high participation from residents and faculty. Future QI projects should engage more staff and evaluate patient perceptions. A limitation was that the charts selected postintervention were not the same as preintervention. The patients whose charts were selected preintervention may not have been due, or may have completed a follow-up visit, when postintervention data were collected. Since our goal was to determine improvements on the clinic level, we do not believe this affected our results or the residents’ ability to learn from this activity. Another limitation was that we only conducted follow-up data once. Due to a lack of resources and changes in QI project priorities (eg, focus on diabetes 1 year later), we were unable to conduct additional follow-ups. Future QI activities should include longitudinal approaches.
This experiential learning activity equipped residents with tools necessary for implementing QI projects after residency. In conjunction with hands-on education in QI methodology, residents also gained experience working in interprofessional teams, applying guidelines to practice, and utilizing EMRs to better understand patient health. We believe this experiential learning activity provided a valuable framework for residents to gain hands-on QI experience while impacting health outcomes among their patients. Our focus on women’s preventive services can be modified to meet other health metrics.
Presentations: This study was presented at the following venues:
North American Primary Care Research Group’s 42nd Annual Meeting, New York, NY. November 2014.
JPS Hospital Annual Research Day, Fort Worth, TX. June 2015.
Institute for Healthcare Improvement 16th Annual International Summit, Grapevine, TX. March 2015.
Society of Teachers of Family Medicine Annual Spring Conference. San Antonio, TX. May 2014.
Dallas Chapter of the Texas Academy of Family Physicians’ Poster Research Contest. Dallas, TX. May 2014.
- ACGME Program Requirements for Graduate Medical Education in Family Medicine. Accreditation Council for Graduate Medical Education. 2019. Accessed June 6, 2020. https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/120FamilyMedicine2018.pdf?ver=2018-06-15-112624-307
- Wong BM, Etchells EE, Kuper A, Levinson W, Shojania KG. Teaching quality improvement and patient safety to trainees: a systematic review. Acad Med. 2010;85(9):1425-1439. doi:10.1097/ACM.0b013e3181e2d0c6
- Starr SR, Kautz JM, Sorita A, et al. Quality improvement education for health professionals: a systematic review. Am J Med Qual. 2016;31(3):209-216. doi:10.1177/1062860614566445
- Brown A, Atchison K, Hecker K, Kassam A. A tale of four programs: how residents learn about quality improvement during postgraduate medical education at the University of Calgary. Teach Learn Med. 2020:1-17. doi:10.1080/10401334.2020.1847652
- Goldman J, Kuper A, Baker GR, et al. Experiential learning in project-based quality improvement education: questioning assumptions and identifying future directions. Acad Med. 2020;95(11):1745-1754. doi:10.1097/ACM.0000000000003203
- Smith F, Alexandersson P, Bergman B, Vaughn L, Hellström A. Fourteen years of quality improvement education in healthcare: a utilisation-focused evaluation using concept mapping. BMJ Open Qual. 2019;8(4):e000795. doi:10.1136/bmjoq-2019-000795
- Tudiver F, Click IA, Ward P, Basden JA. Evaluation of a quality improvement curriculum for family medicine residents. Fam Med. 2013;45(1):19-25.
- Hall Barber K, Schultz K, Scott A, Pollock E, Kotecha J, Martin D. Teaching quality improvement in graduate medical education: an experiential and team-based approach to the acquisition of quality improvement competencies. Acad Med. 2015;90(10):1363-1367. doi:10.1097/ACM.0000000000000851
- Kirkman MA, Sevdalis N, Arora S, Baker P, Vincent C, Ahmed M. The outcomes of recent patient safety education interventions for trainee physicians and medical students: a systematic review. BMJ Open. 2015;5(5):e007705. doi:10.1136/bmjopen-2015-007705
- Screening for Breast Cancer: U.S. Preventive Services Task Force Recommendation Statement. Ann Intern Med.2009;151:716-726. [Epub ahead of print 17 November 2009]. doi:10.7326/0003-4819-151-10-200911170-00008
- Moyer VA. Screening for cervical cancer: U.S. Preventive Services Task Force recommendation statement. Ann Intern Med. 2012;156(12):880-91, w312. doi:10.7326/0003-4819-156-12-201206190-00424.
- U.S. Preventive Services Task Force. Screening for colorectal cancer: U.S. Preventive Services Task Force recommendation statement. Ann Intern Med. 2008;149(9):627-637. doi:10.7326/0003-4819-149-9-200811040-00243
- Edwards JPS, Schultz L, Erickson N, Dykstra D. Horizons: The Dallas County community health needs assessment. Dallas, TX: Dallas County Health and Human Services and Parkland Health and Hospital System; 2012. Accessed June 6, 2020. https://www.parklandhospital.com/Uploads/Public/Documents/PDFs/Community/Dallas%20County%20Community%20Health%20Needs%20Assessment%202013%20Horizons.pdf
- State Cancer Profiles. National Cancer Institute. 2016. Accessed June 6, 2020. https://statecancerprofiles.cancer.gov/index.html
- Healthy People 2020. Office of Disease Prevention and Health Promotion, United States Department of Health and Human Services. Accessed July 22, 2021. https://www.healthypeople.gov/.
- Singh MK, Einstadter D, Lawrence R. A structured women’s preventive health clinic for residents: a quality improvement project designed to meet training needs and improve cervical cancer screening rates. Qual Saf Health Care. 2010;19(5):e45. doi:10.1136/qshc.2009.033274
- Feldman J, Davie S, Kiran T, Measuring and improving cervical, breast, and colorectal cancer screening rates in a multi-site urban practice in Toronto, Canada BMJ Open Quality 2017;6:u213991.w5531. doi:10.1136/bmjquality.u213991.w5531.
- Lee SJC, Inrig SJ, Balasubramanian BA, et al. Identifying quality improvement targets to facilitate colorectal cancer screening completion. Prev Med Rep. 2018;9:138-143. doi:10.1016/j.pmedr.2018.01.004
- Beaber EF, Kim JJ, Schapira MM, et al; Population-based Research Optimizing Screening through Personalized Regimens Consortium. Unifying screening processes within the PROSPR consortium: a conceptual model for breast, cervical, and colorectal cancer screening. J Natl Cancer Inst. 2015;107(6):djv120. doi:10.1093/jnci/djv120