Exploring Change After Implementation of Family Medicine Residency Curriculum Reform

In 2010, the College of Family Physicians of Canada (CFPC) introduced the Triple C Competency-based curriculum (Triple C), a competency-based medical education (CBME) approach to guide Canadian family medicine (FM) residency training. Triple C is designed to foster an environment that would enable learners to acquire FM-specific competencies through learning experiences provided in FM contexts.1 The “three Cs” of Triple C align with the mission of the CFPC to reform training in order to graduate family physicians who are ready to begin the practice of comprehensive FM in any community in Canada.1-3 Triple C promotes training environments that are Comprehensive (from intrapartum to palliative/end-of-life care); allow for Continuity (care across the lifecycle); and are Centered in FM (learning and training reflect actual practice experiences). The notion of readiness to begin the practice of comprehensive FM is in keeping with a competency-based approach to learning that emphasizes ongoing attainment of skills and competencies, with the achievement of key milestones during and at the end of residency training.

One impetus for the development of Triple C was concerns raised recently regarding an emerging trend of FM graduates moving away from comprehensive FM toward more focused and narrowed practices.4-6 As such, one of the intended outcomes of Triple C is to encourage the practice of comprehensive FM among graduates. Another intended outcome of Triple C relates to the CFPC’s social accountability mandate, which is to enhance access to care for underserved populations, such as those in rural and remote communities in Canada.

When Triple C was introduced in 2010, a comprehensive program evaluation plan was launched to explore if intended outcomes were achieved.7 As part of the program evaluation plan, the Family Medicine Longitudinal Survey (FMLS) was developed to explore the perceptions and practice intentions of FM residents entering residency, exiting residency, and 3 years into practice.7 To date, preliminary findings from a small set of residency programs that piloted the FMLS have been published.8 This paper shares encouraging results with a trend toward increases in self-reported intention to practice comprehensive FM; however, it is important to note that pre-Triple C baseline data, as a benchmark for comparison, were not collected.

Critics of Triple C question whether change has occurred in response to implementation of this CBME approach, or if other factors are responsible. The paucity of deliberately collected data pre-Triple C continues to pose significant challenges for all stakeholders involved in the Triple C discussion since traditional research and evaluation approaches that would normally provide insight into outcomes of this curriculum change are not possible. For example, this criticism primarily stems from use of methods other than randomized controlled trials and subsequent challenges when researchers assert causation. While there is a vast array of research methods available and feasibly implemented in evaluations of complex interventions, such as curricular reform, a shift from asserting causation to identifying association and contribution is necessary.

One method to explore association and contribution of educational innovations in the absence of baseline comparative data is to look for proxies for baseline data that already exist. In this study, we used historical data about the practice intentions of FM residents retrieved from Canada’s National Physician Survey (NPS), given to Canadian physicians and residents from 2004 to 2014. Specifically, we compared findings from similar survey questions on NPS 2007 and NPS 2010 (pre-Triple C) with responses from FM residents on the FMLS in 2015 (post-Triple C). There are sufficient parallels between the questions asked on the NPS and the questions asked on the FMLS to justify the use of the NPS as a proxy for preimplementation baseline data in examining the outcomes of Triple C.

The goals of this paper are twofold. The first is to provide results on outcomes from the implementation of a CBME curriculum by comparing responses from graduates of FM residency programs across Canada pre and post-Triple C implementation, specifically in the area of their intentions to practice across a broad range of clinical domains. The second goal of this paper is to offer a case study of the feasibility of using historical data acting as a proxy for baseline data in comparing outcomes pre- and postimplementation of a CBME program in the absence of deliberately collected baseline data.

A retrospective secondary data analysis of cohorts of graduating trainees (R2) from 2007 and 2010 (pre-Triple C), and 2015 (post-Triple C) was used to examine possible changes in intentions to practice as reported by family medicine residents over time.

1. National Physician Survey (NPS). The NPS is an anonymized, voluntary, self-report measure administered to Canadian physicians, residents, and medical students.9 It was produced in collaboration with the CFPC, Royal College of Physicians and Surgeons, and the Canadian Medical Association to gain information about physician services in response to societal needs.9 The NPS survey captured information on practice intentions and career paths of residents across all specialties, including FM, at different intervals from 2004-2014; however, a collective decision by the three lead organizations was made to discontinue the use of the NPS in 2015. Data from two iterations of NPS (2007 and 2010) containing content on resident perspective were analyzed in this study.10,11

2. Family Medicine Longitudinal Survey (FMLS). The FMLS is a pan-Canadian anonymized, self-report measure administered by the CFPC as part of the Triple C’s program evaluation plan, created specifically to explore curriculum outcomes of resident experiences in FM training.7 Each of the local ethics boards at participating residency programs gave ethical approval to implement the survey as part of a longitudinal study/program evaluation plan. The creation of the FMLS was guided by knowledge of the NPS and included a subset of similar questions. This subset includes questions related to the exploration of resident learning experiences and practice intentions. In 2015, the first pan-Canadian cohort of exiting FM residents, from 15 of the 17 residency programs, participated in the FMLS.

Comparable questions from both the NPS (2007 and 2010) and FMLS (exit 2015; Table 1) were identified and responses from FM residents collated from each.10,11 Due to a lack of consistency of questions used in the different iterations of the NPS, only questions with similar themes were included for comparison. To enhance comparability between NPS 2007, NPS 2010, and FMLS 2015, only responses from the 15 family medicine residency programs that participated in both surveys were included.

The NPS used dichotomous measures (yes/no) to collect information on resident intentions to practice, while the FMLS used a 5-level Likert scale (strongly disagree to strongly agree). As such, FMLS data were aggregated to create a similar dichotomous response format. More specifically, in FMLS, both “strongly disagree” and “disagree” were categorized as “no” and both “strongly agree” and “agree” were categorized as “yes.” “Neutral” responses were divided equally with half assigned “yes” and the remaining half-assigned “no” responses. Missing data from any of the surveys in the form of noresponses were not included in the analyses.

We retrieved the total number of R2 FM residents for each cohort directly from the respective databases for NPS 2007 and FMLS 2015. The total number of R2 FM residents was not available for the NPS 2010 survey. By referring to the Canadian Residency Matching Service survey (CARMs), which provides information on total numbers of residents matched to FM programs,14 we produced an estimate of the number of R2 residents for the 2010 iteration of the NPS. We calculated demographic data and practice intentions of residents in NPS 2007, NPS 2010, and FMLS 2015, first using descriptive statistics, then comparing between cohort years using a series of Pearson χ2 tests. A modified significance value was set at P<0.01 to account for multiple comparisons.

Exit responses of FM residents (R2) were the focus of analysis in NPS 2007, NPS 2010, and FMLS 2015. Table 2 illustrates response rates, as well as the number of FM residents who responded from the 15 residency programs, including mean age at completion of training. In comparing the two surveys and iterations, findings reveal a significant decrease in the number of residents expecting/having children from NPS 2007 to NPS 2010 and FMLS 2015 data (Table 2; P<0.01). Similar proportions of FM residents grew up in inner city/urban/suburban environments and in small town/rural environments, but an increased trend of FM residents from urban centers may be emerging; however, current results are not statistically significant. Overall, the demographics of FM residents across surveys were similar.

Significant differences were noted when comparing the self-report practice intentions of FM residents across Canada from NPS 2007, NPS 2010, and FMLS 2015 (Table 3). When comparing NPS 2007 and NPS 2010 data, significantly fewer FM residents in NPS 2010 indicated an intention to provide palliative care and in-patient hospital care as part of their future FM practices (Table 3; P<0.01). Among responses from the FMLS 2015, significantly more FM residents reported intention to provide palliative care as compared to FM residents responding in NPS 2007 (Table 3; P<0.01). Based upon FMLS 2015 data, a significant increase was seen in FM residents compared with NPS 2010 who shared an intent to provide palliative care, intrapartum care, in-patient hospital care, care in the home, and practice in rural settings upon exiting residency (Table 3; P<0.01).

The results presented in this paper provide a description of practice intentions of FM residents before and after implementation of Triple C, an approach to CBME, using existing pan-Canadian data. The use of historical data as a proxy for preimplementation baseline data was pragmatic and relatively novel in the field of medical education research, mitigating the current challenges of the absence of purposive baseline or pre-CBME implementation data. While it is clear that two different survey tools were used, a comparison of responses from similar questions was conducted with a similar demographic subset of FM residents from the same 15 of 17 Canadian FM residency programs.10,11 As such, the findings from this comparison are relevant.

Given that one of the goals of the Triple C is to design curriculum and clinical experiences to encourage graduating residents to consider comprehensive care, it was important to explore the breadth of clinical domains that graduates intend to include in their future practices. These findings may support a contribution by Triple C as an influencing factor in graduates’ intentions to practice comprehensive care, rather than the observed trend toward more focused FM practice. Findings in this study suggest that over time there is a trend for FM graduates to report their intentions to include a broader range of clinical domains in their future practice. It is important to note that when graduates consider inclusion of more clinical domains in future, a broader scope of FM practice is assumed to be more likely for those residents. Our interpretation is strengthened by findings from FMLS 2015 exit data, which indicate that upon completion of residency training, over 70% of residents are confident to begin the practice of comprehensive family medicine in any community in Canada.15

Based on the comparison between NPS 2010 data and FMLS 2015 data, fewer residents pre-Triple C (NPS 2010) indicated the intention to provide palliative care and intrapartum care, and home care in their future practices upon graduation. This was a decrease from the NPS 2007 data, suggesting that a trend of narrowing scope of practice among family medicine residency program graduates had begun. Most residents responding to the 2010 NPS survey would have started their residency in 2008. These findings align with Canadian literature during this time calling for more intentional and improved teaching of intrapartum care to FM residents, as well as advocacy for more palliative care education.16-18 The trend of a narrowing scope of practice was less evident in the FMLS 2015 data. Indeed, the results presented here illustrate a positive shift in the 2015 data of intentions to practice in-home and palliative care, which is encouraging since population statistics indicate the number of elderly individuals in the Canadian population continues to grow.27 Some evidence suggests that learning experiences in comprehensive domains, settings, or populations of practice may increase confidence and motivation, and may be associated with a higher likelihood of practicing comprehensive FM upon graduation.19

Another goal anticipated through the development of the Triple C curriculum was to influence equitable distribution of FM graduates, particularly in rural and remote Canadian communities. The FMLS 2015 data showed an increase in the number of exiting residents intending to practice in rural communities,15 a finding that corresponds with existing literature highlighting increasing numbers of FM graduates choosing to practice in underserved communities.29-32

Using historical data as a proxy to mitigate the lack of pre-Triple C baseline data was a novel and pragmatic solution to creating a comparator with post-Triple C implementation data. Through our analyses comparing the three surveys (NPS 2007, NPS 2010, FMLS 2015), we have illustrated that there are no significant differences in the demographics of respondents, aside from having/expecting children. The similarities between the cohort years despite the use of different surveys contributes to the strength of the results presented here. Further, we have taken a methodologically sound approach to identifying data that offers a proxy for benchmark information captured from FM residents prior to the implementation of the Triple C curriculum. This approach allowed us to determine preliminary, but compelling, findings of the contribution of Triple C to resident practice intentions.

As with any study, there are limitations. The use of a randomized controlled trial is often suggested as the gold standard methodology for rigorous and objective research.33-35 In the case of curriculum reform and particularly at a pan-Canadian level, implementing Triple C (or any other CBME approach) using this research design was not feasible. As such, we suggest that there is an opportunity to use historical data as a proxy for baseline data given that no purposive data collection was available pre-Triple C implementation. Although the use of survey data has provided insight into the practice intentions of FM residents, these findings may not be reflective of all FM residents. The practice of FM varies across Canada, often with regional-specific politics, locally-driven norms, and diverse perceptions of FM as a discipline; such variations may play important roles in influencing practice intentions. Opportunities to use qualitative methods to explore reasons for choice of practice scope and more in-depth inquires related to the influence of norms and attitudes are needed.

Another potential limitation relates to the use of self-report ratings without availability of other measures for triangulation or confirmation of findings. There is evidence that truthfulness and test-retest reliability is high when individuals are surveyed through paper or online format.36-38 Future comparisons are needed to identify patterns and trends from data emerging from the FMLS 2016 to FMLS 2020 exit surveys. In addition, FM graduates 3 years into practice will be invited to participate in a third FMLS survey in 2018. This information will give further insight into the relationship between resident-reported practice intentions at the end of training and actual practice.

This study’s findings provide information on the practice intentions of graduates from family medicine residency training before and after the implementation of a CBME innovation, the Triple C curriculum. The use of previously collected survey data as a historical benchmark is a pragmatic and novel approach to comparing pre- vs postimplementation outcomes in the absence of deliberately captured baseline data. This comparison to historical data allowed for identification of key changes in specific outcomes across FM residency programs from 2007 to 2015. The findings presented here should not be viewed with the lens that Triple C was the only factor causing changes in practice intentions of FM residents; rather, that Triple C may have contributed to the changes observed. More research is needed to explore impact and to look for mediating or moderating factors influencing physician practice intentions. In an era when quality is a key focus for health care delivery, and equitable access to health care services is an important quality indicator, the family physician’s ability to provide comprehensive care that aligns with the ever-changing needs of the community is imperative.