Implementing Competency-Based Medical Education in Family Medicine: A Narrative Review of Current Trends in Assessment

Competency-based medical education (CBME) is an outcomes-based approach to the design, implementation, assessment, and evaluation of an educational program. It uses an organized framework of competencies, that ultimately may lead to better care for patients.1 By definition, CBME demands a robust and multifaceted assessment system that embraces continuous evaluation of trainees.2,3 Accurate assessment of competence, or the “meaningful assessment of competence,”4 requires continuing, serial, and direct observation of workplace behavior and monitoring of progress based on feedback.3,5

Since the implementation of effective CBME depends on mounting high-quality assessment practices, building a coherent and integrated system of assessment across the continuum of training to practice is critical.6 As such, the developmental progression of competencies must be assessed at all stages of the learning process, including continuing professional development (CPD).6-11 Yet much of the recent discussion around CBME implementation revolves mostly around residency programs,1,12-15 including family medicine residency programs in Canada16-19 and the United States.20-23 Our recent scoping review24 on the current state of CBME implementation in family medicine residency and CPD programs has also demonstrated the paucity of studies on the applicability or relevance of CBME to physicians in practice.

Family medicine was one of the earliest adopters of CBME framework in Canada and the United States with its programs being implemented into residency training on a nationwide scale.12,16,22,25,26 The challenge is that the studies that have focused on examining competency-based assessment tools (eg, identifying challenges, providing criteria, basic principles, and guidance for good assessment),4,6,11,27-30 have not paid sufficient attention to the overall implementation of assessment practices, particularly in family medicine. In other words, there are no studies that have thematically and systematically synthesized the current family medicine literature pertaining to the overall CBME assessment implementation and identified gaps in existing knowledge.

The purpose of this review was to synthesize the findings of studies spanning the last 2 decades that examined competency-based assessment methods used in family medicine residency and CPD, in order to identify gaps in current practices in implementing competency-based assessment tools. Given the breadth of this topic, we specifically focused on the following research questions:

1. What competency frameworks were described in the included studies?
2. What assessment methods and assessment systems were discussed?
3. Was the purpose of the assessment formative, summative, or a combination?
4. What types of educational outcomes, outcome frameworks, and outcome levels were described in the family medicine literature?

For the purpose of this paper, we focused primarily in CBME for use in CPD in family medicine. However, little has been written about practical CBME implementation within CPD in any specialty,11,31,32 particularly in family medicine.33 Therefore, we also searched the residency training in family medicine literature to identify potentially pertinent and transferable findings to CPD. The findings will facilitate our understanding of how CBME implementation practices in residency training might inform the design and operationalization of CBME implementation strategies in CPD.

We used a narrative review approach34-40 to thematically synthesize the current literature and identify gaps in existing knowledge. The present review sets about to identify current studies on competency-based assessment implementation in family medicine residency and CPD.

To avoid potential pitfalls associated with narrative reviews, such as selection bias, lack of diversity in sources, or drawing conclusions based more on opinion than data, the preparation of narrative reviews must apply the methodological rigor of systematic reviews.35,37,40-42 Specifically, we employed a systematic approach to both steps: (1) selecting studies to be included, and (2) extracting information from primary articles.35,43,44 A systematic approach implies grouping and analyzing sources with similar findings and/or the same level of evidence. This can be done by placing data from the selected sources in tables and analyzing the data in the main body, without duplicating information.42

A modified form of narrative review, as described by Popay et al,45 Ferrari,37 and Baethge et al,40 was adopted whereby data extraction enabled synthesis of key data, while also allowing rich narrative description.37 We adapted the the steps involved and outlined below from Shaw et colleagues,35 and our research team modified them.

A preliminary search of the literature was undertaken to see what work in the area of interest had been published and to verify that no similar review had been published already.35,44,46 Our scoping search revealed no existing reviews of the state of knowledge regarding CBME assessment implementation in family medicine residency and/or CPD.

To improve the method of literature selection and to reduce the risk of suboptimal reporting,37 our modified search strategy employed components from a systematic review methodology (PRISMA) which involves screening titles and abstracts as well as data extraction techniques.47 The strategy (selecting the databases and defining the inclusion and exclusion criteria and the search terms) was developed by authors S.K., N.D., and the Health Sciences librarian A.H. The initial search (from 2000 through April 30, 2017) was conducted on April 28, 2017, and a follow-up search was done on May 5, 2020, using the following five electronic databases: MEDLINE, ERIC, PsycINFO, Embase, and EdSource. Additionally, we searched government-related and relevant professional organizations’ websites. As the Accreditation Council for Graduate Medical Education (ACGME) Outcome Project started in 1999, we restricted our search to the literature published after 2000 to capture information that was relevant to CBME in family medicine. We included all original research, review articles, editorials/commentaries, and regulatory papers.48 The full search strategy and the specific search terms, that were identified through input from the research team and an academic librarian, are provided in the Supplemental Digital Appendix (https://journals.stfm.org/media/3555/hendry-supp-appendix1.pdf).

Studies that fulfilled the following criteria were included:

1. focused exclusively on family medicine residency or CPD programs. Any undergraduate medical education article was excluded;
2. discussed CBME;
3. were American or Canadian in origin;
4. published in English;
5. available in a full-text version (the articles were requested through the library if immediate full-text versions were not discovered).

The first reviewer (N.D.) screened and read the titles and abstracts of all identified studies.43,44 Studies that fell under the exclusion criteria were removed, as were duplicates and studies that did not meet the inclusion criteria. N.D. screened full-text documents and excluded those not relevant. Questionable cases were read by the second reviewer (S.K.), and a joint decision was reached on whether to include them or not for further review.

Step 1. We developed a first version of a standardized data extraction form based on the literature.4,5,18,49-55 We piloted the data extraction form with all team members three times.

Step 2. A final data extraction form consisted of two parts. We created the first part to draw key demographic characteristics from the articles (publication year, publication type, study design, year, author, country, target audience; see Table 1). The second part of the extraction form included the coding concepts developed through a review of the published literature and revised in consultation with the research team. The definitions for these coding concepts (competency framework, assessment method, assessment purpose, outcomes model, type of educational outcome, and level of outcome) are shown in Table 2.4,5,16,18,27,56-82 N.D. collected the data on general characteristics of the articles. Furthermore, several reviewers independently extracted the key elements of CBME assessments; the discussion method was used first and if disagreements could not be resolved, further checks were made. If there were any ambiguous items, the principle investigator (S.K.) reviewed the article, and made the final decision. We tabulated and analyzed the data.

We used an inductive approach36,44,83,84 to reflect upon the “landscape of events”85 of competency-based assessment in family medicine residency and CPD in a North American context. The themes identified were strongly linked to the raw data and were not necessarily related to the research questions posed.34,86 The expert/research team members discussed the results of the review to gain an overall understanding of the trends and nature of competency-based assessments evident in family medicine residency and CPD literature during the last 2 decades.36,84

The original search yielded 1,222 potentially relevant citations. After deduplication and relevance screening, 185 citations met the eligibility criteria based on title and abstract and the corresponding full-text articles were procured for review. After data characterization of the full-text articles, we retained 37 articles in the analysis (Figure 1), with 148 being excluded for one of the following reasons: no evidence of CBME assessment tools description (n=62); no evidence of the CBME concept used (n=30); irrelevant to family medicine field (n=27); oral/poster presentations (n=17); not a Canadian or US article (n=7); and undergraduate medical education (n=5).

The 37 studies5,18,20,33,55,60-63,66,67,69,87-111 were published between January 2000 and May 2020, with nearly half of the articles (17/37; 46%) published within the past 5 years (2015-2020). The general characteristics of the articles included in this study are reported in Table 1. The majority (21/37; 57%) of all studies originated from the United States. Research articles (24/37; 65%) and commentary/reflective papers (8/37; 22%) comprised most documents; articles characterized as editorial (2/37; 5%), review (2/37; 5%), and regulatory (1/37; 3%) were underrepresented. Among 24 studies eligible for classification by a type of research paradigm, 50% (12/24) used a quantitative approach, 38% (9/24) used a qualitative approach, and the remaining used mixed methods (3/23; 12%).

Almost all studies (36/37; 97%) were identified in published literature. Only one article was found in the gray literature; the article was published on the College of Family Physicians of Canada (CFPC) website. The articles were published in 13 different journals, although two-thirds of the papers (25/37; 67%) were concentrated in three journals. These included 37% (14/37) of studies published in Family Medicine, 19% (7/37) in Canadian Family Physician, and 11% (4/37) in Academic Medicine. More than half of the studies (23/37; 62%) reported residents being their target population followed by both faculty and residents (11/37; 30%), and family physicians (3/37; 8%).

Table 2 provides the definitions and summarizes coding frequencies of the eight coding concepts across the data set. The ACGME/ABMS framework was discussed more frequently (21/37; 57%), followed by the CanMEDS-FM (14/37; 38%) and Triple C frameworks (2/37; 5%). While the majority of the articles (20/37; 54%) focused on individual assessment methods, the remaining studies discussed assessment systems (13/37; 35%) as reviewed in Table 2. Half of the studies (20/37; 54%) focused solely on formative purpose of assessment, but 13 (35%) papers discussed both formative and summative assessments. None of the studies concentrated exclusively on summative assessment.

The coding concept “outcomes model” was retrieved from one source only. We mapped the different types of educational outcomes (highlighted in Table 2) to the assessment frameworks of Moore et al.,50 Miller,74 and Kirkpatrick and Kirkpatrick,75 as suggested by Price et al.73 The majority (17/37; 46%) of studies targeted program-level followed by individual-level outcomes (12/37; 32%). Eight papers (22%) described both levels of outcomes. We grouped all articles by a stage of assessment method implementation, ranging from the stage “development of assessment methods” to the stages “implementation and initial evaluation of assessment methods,” and “outcome and impact evaluation of assessment method.” Nearly half of the studies (18/37; 40%) described assessment tools at their initial stage of implementation (eg, defining, designing, and planning of assessment instruments), while the other half (19/37; 51%) discussed partially implemented (5/37; 13%) or fully implemented and evaluated assessment methods (14/37; 38%).

Table 2 also highlights the frequency of the coding concepts among the three categories of the articles: residency articles, CPD articles, and CPD/residency articles. Direct observation was the only assessment method described in CPD articles, while CPD/residency studies more often discussed multiple assessment tools (3/11; 27%). Yet, residency articles equally focused on the two assessment methods: direct observation (4/37; 17%) and competency-based achievement system (4/37; 17%). The individual level of educational outcomes was used more often in CPD/residency (6/11; 55%), whereas the program level outcomes were most common in residency (12/23; 52%) and CPD studies (3/3; 100%). Finally, all CPD and almost half of residency studies (11/23; 48%) described assessment tools at their initial stage of implementation (eg, defining, designing, and planning of assessment instruments). In contrast, the majority of CPD/residency papers (5/11; 46%) discussed fully implemented and evaluated assessment methods.

Through the process of iterative reading and discussion of the literature by the research team, three broad themes emerged: ways to improve assessment methods, assessors’ needs and challenges, and learners’ needs and challenges. A summary of the themes and subthemes along with frequency counts are presented in Table 3.

The purpose of this review was to synthesize the findings of studies spanning the last 2 decades that examined the competency-based assessment methods used in family medicine residency and CPD, in order to identify gaps in current practices in implementing CBME assessments. Our analysis shows that there is a very small body of published work on competency-based assessments in family medicine residency and CPD.

The following discussion highlights the three major implications of our analysis: (1) trends in competency-based assessment, (2) challenges to implementing competency-based assessment, and (3) key elements for supporting competency-based CPD.

The paucity of articles (14/37; 38%) reporting reliability of assessment methods, intention to use them, and impact of assessment instruments on residents and faculty is not surprising, given the lack of frameworks that define a CBME program,112,113 inconsistency around the CBME language,114-116 the difficulties inherent in assessing competence,5 and a limited focus on a broad range of issues related to fidelity of CBME impementation.24 Articles that focused solely on formative assessment were most prevalent among all three categories of articles (Table 2). This may reflect that CBME programs are paying increasing attention to competencies beyond knowledge. It may also reflect the use of formative rather than summative in-training examinations in residency programs.

The most common types of educational outcomes were performance (23/37; 62%) and competence (8/37; 22%; Levels 5 and 4 of Moore et al’s pyramid50), suggesting that a portfolio of formative assessment techniques has been shown to be effective at measuring competence and performance within any proposed outcomes-based framework.50,100 In contrast, we found no studies on the assessment of patient and community health outcomes in our search, which may be explained by multiple challenges related to their measurement (eg, lack of available data, compounding due to multiple interventions).117 While summative assessment techniques are well established and have been proven effective for measuring knowledge, these assessment methods were less commonly identified in our review (13/37; 35%). Given program directors’ need to sign off on their residents’ preparedness to enter practice, this percentage (35%) appears low from a residency perspective. From a CPD perspective, this is rather high, because outside of board certification/maintenance of certification, most CPD programs were not designed for summative evaluation, but rather to promote continuous improvement of practice.

Based on the thematic analysis of the literature, we identified that the Family Medicine educational community faces several challenges with CBME assessment (Table 3). Those challenges reflect some of the current trends in the general medical educational literature regarding competency-based assessment.4,11,118,119

We found that the most popular concern for assessors was the need for standardised training for assessment. There were calls for faculty development training to focus on the effective use of field notes and observation, better interrater consistency and reliability, training on how to effectively teach and evaluate the intrinsic roles (communicator, collaborator, leader, health advocate, scholar, and professional), and training on being faculty advisors. These results are in line with many studies identifying gaps in skills required by faculty to consistently assess competencies as part of the redesign of residency training programs.28,120,121 Additionally, faculty overload was mentioned as one of the major challenges, particularly the ability to integrate assessments within workflow in a clinical setting, which is in agreement with previous studies.27,122 In contrast, the most common concern for learners was the need for better constructive feedback, followed by the need for residents’ [to receive] better orientation vis-à-vis expectations and processes prior to the start of the sessions. It has been shown that learners actively seek out critical feedback to help them accomplish the competency goals in order to advance through the stages of formal education.123 The theme of “ways to improve assessment methods” included several subthemes; “having end-user input for the evaluation of the assessment tools” was the most frequently-cited topic. Feedback from residents, advisors, program directors, program and site administrators, and off-service preceptors was suggested as a means of improving the assessment tools. The active engagement of learners in their own assessment and feedback from faculty are both important, as they ensure the ultimate acceptability of an assessment tool by the key stakeholders.5,124-127

Finally, despite continuing discussion about the role of CPD in building CBME assessment across the continuum of training to practice,4,6,31,128 the main gap in the educational literature we uncovered was a total absence of studies related to the ongoing assessment of practicing physicians. This seems to be in line with the overall lack of formal evaluation studies in family medicine CPD reported previously by our group.33 Likewise, our recent scoping review has revealed a shortage of scholarships on CBME implementation practices within family medicine CPD.24 One elephant in the room is the disjointed nature of contemporary medical education129—the silos that exist between undergraduate, graduate, and continuing medical education. Given that the continuum of learning ideal has yet to be realized, we advocate for closer collaboration among stakeholder organizations responsible for each level of medical training in order to design a comprehensive system of assessment.6,33,129 Additionally, the integration of efforts across the four domains (continuing education, knowledge translation, patient safety, and quality improvement) should aim to develop a systematic approach to competency assessment across the continuum of health professions education and practice.6,33,129 Without a structural basis for competency assessment across the continuum, the difficulties of evaluating competency-based assessment efforts will remain. We suggest that for CBME to be successfully applied to practice, strategies are needed to consistently integrate novel approaches to learning (direct observation, simulation, audit and feedback, multisource feedback, educational outreach visits, etc). These new learning activities (which are both assessment activities and educational interventions) can provide physicians with ongoing assessment and feedback leading to clinical behavioral change.31 Unlike residents, physicians after the completion of training often do not have a formal system that is designed to provide personal support for learning and improvement. Recent recommendations for maintenance of certification systems to evolve systems of CPD to support learning and continuous improvement of practice are strategic attempts to address this specific concern.130,131 Traditional continuing medical education programs, the primary formal educational support structure for practicing physicians, is limited in its ability to change performance and patient outcomes. Practicing physicians need access to trusted practice data with the opportunity to review their data with a peer, coach, or mentor to enable identification of (and minimize resistance to) needed practice changes.132-135 Physicians will need access to resources and an infrastructure to support and motivate them to sustain practice change.136 While CBME is primarily directed toward individuals, improving health outcomes will likely require a team-based strategy along with educational and clinical care systems in order to provide the conditions enabling continuous improvement.

This is the first review study to examine the nature and trends of competency-based assessment methods in family medicine. Its strengths include the assembly of key content and methodological experts from diverse backgrounds.35-39 We acknowledge several limitations of this review. Our review was based on a small set of articles, which speaks to the limited number of publications in the field. In addition, the shortage of CPD articles restricted our ability to compare between residency and CPD studies, so there may be interesting differences or similarities among the two stages of the learning continuum. Lastly, although we sought to search multiple databases including the gray literature, the scope of the search was limited to articles in English published in Canada and the United States, which have similar residency programs in family medicine.137

In this narrative review, we attempted to inform future approaches and research by analyzing and synthesizing the findings of publications that described CBME available assessment tools in family medicine residency and CPD. Our analysis shows: (1) a very small body of published work currently exists around competency-based assessments in family medicine; (2) a lack of studies on assessment methods among practicing physicians (a gap in ongoing assessment in clinical practice); and (3) common themes that may offer insights into how current assessment practices might be improved to ensure alignment with modern conceptions of health professional education for the ultimate goal of improved health care. We recommend that those involved in family medicine education should strive to systematically evaluate and publish their CBME activities, including assessment-related content and evaluations. In addition, one of the important avenues for future research should illustrate residents’ and faculty’s perception of how new approaches to curriculum design and evaluation are impacting their learning/teaching. Finally, we suggest that in building a coherent and integrated system of assessment, evaluation of the key contextual factors across the continuum of education to practice is of increasing importance in the field of CPD.