BACKGROUND AND OBJECTIVES: While family medicine has been one of the first specialties to implement competency-based medical education (CBME) in residency, the nature and level of its integration with continuing professional development (CPD) is neither well understood nor well studied. The purpose of this review was to examine the current state of CBME implementation in family medicine residency and CPD programs in the North American education literature, with the aim of identifying implementation concepts and strategies that are generalizable to other medical settings to inform the design and implementation of residency training and CPD.
METHODS: Using an Arksey and O’Malley six-step framework, we searched five online databases and the gray literature over the period between January 2000 through April 2017. We included full-text articles that focused on the key words CBME, residency, CPD, and family medicine.
RESULTS: Of the articles reviewed, 37 met the inclusion criteria and were selected for full review. Eighty six percent of included articles focused on foundation elements related to designing competency-based curriculum and assessment strategies rather than program evaluation or other outcome measures. Only 19% of the articles were related to CPD that focused only on the implementation at the program and/or institution/organization levels.
CONCLUSIONS: Given that the implementation of CBME is in its relative infancy, the pattern of implementation activities described in this scoping review reflected a limited focus on a broad range of issues related to fidelity of implementation of this complex intervention.
Competency-based medical education (CBME) is an outcomes-based approach to the design, implementation, assessment, and evaluation of an educational program using an organized framework of competencies, which ultimately may lead to better care for patients.1 Promoting CMBE across the medical education continuum from undergraduate to residency training and continuing professional development (CPD) is envisioned to improve quality and patient safety, considering all stages of professional development contribute to physicians’ knowledge, skills, attitudes, behaviors, and competencies.1-3 Much of the recent discussion around CBME implementation focuses on residency programs,1,4-8 including family medicine residency programs in Canada9-15 and the United States.16-19
Family medicine is uniquely positioned to improve the health of patients and advocate for health equity and patients’ social well-being.20 As family medicine gains importance in the delivery of health care, it is imperative that family physicians are trained in accordance with international educational trends and societal expectations.9,20-22 In order to enhance the quality of residency medical education, the College of Family Physicians of Canada (CFPC) transitioned all family medicine programs to a competency-based Triple C Curriculum.9,23 The Triple C Curriculum emphasizes enabling competencies within seven roles: medical expert, communicator, collaborator, health advocate, scholar, manager, and professional. In the Triple C Curriculum Report, 11 recommendations to change family medicine residency training included the following statement: “A competency-based approach should be used to guide curriculum development and planning.”9 A year later, the Future of Medical Education in Canada: Postgraduate Education Project24 recommended all residency training in Canada “develop, implement, and evaluate competency-based, learner-focused education to meet the diverse learning needs of residents and the evolving healthcare needs of Canadians.”24 The transition from time-based to competency-based education residency programs was intended to explicitly teach and assess the specific competencies needed for practice.
Similarly, in the United States, in 1998, the Accreditation Council for Graduate Medical Education (ACGME) Outcomes Project was initiated to emphasize educational outcome assessment in residency programs and the accreditation process.25 The ACGME and American Board of Medical Specialties (ABMS) jointly endorsed six competency domains: patient care, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice.26 Similar to residency training, CPD based on competency frameworks is envisioned to promote lifelong learning, practice-based change, and improve clinical outcomes.2,8,27-29 As the CBME education systems evolve, there is a need to understand how CBME implementation practices have been adopted across residency education in order to inform the future design and operationalization of CBME in CPD.
Family medicine was one of the earliest adopters of CBME in Canada and the United States with its programs being implemented into postgraduate training on a nationwide scale.4,9,19,30,31 This scoping review represents a timely foundational summation of the existing evidence around the nature of CBME implementation activities in family medicine. The purpose of this study was to use a scoping review methodology to examine the extent, range, and nature of CBME implementation practices in family medicine residency and CPD programs across Canada and the United States. In this review, we addressed the broad question: “What is the scope and nature of the CBME implementation practices in family medicine residency and CPD in the North American scholarly research?” We aimed to address this broad question through first addressing the following subset questions:
- What is the range and prevalence of CBME implementation practices in family medicine residency and CPD across Canada and the United States discussed in scholarly literature?
- What are similarities and differences in the CBME implementation practices between residency and CPD?
We organized this scoping review32,33 into six stages: (1) identifying the research question, (2) identifying relevant studies, (3) study selection, (4) charting the data, (5) collating, summarizing, and reporting results, and (6) consultation (optional). We increased methodological rigor by making consultation with content experts the first step and an ongoing process throughout the review.
Stage 1: Ongoing Consultation
We circulated the scoping review protocol draft for feedback from experts/stakeholders in three domains: CPD, knowledge translation, and family medicine. These content experts were engaged at four critical points, (1) developing the methodology, (2) reviewing search results, (3) discussing emerging themes and gaps, and (4) reviewing the manuscript.
Stage 2: Identifying Research Questions
The scoping review focused on answering the following two questions:
- What is the range and prevalence of CBME implementation practices in family medicine residency programs and family practices across Canada and the United States?
- What are similarities and differences in the CBME implementation practices between residency and CPD?
Stage 3: Identifying Relevant Studies
A University of Ottawa senior librarian developed our search strategy, and the project team subsequently revised the strategy. We searched five electronic databases (MEDLINE, ERIC, PsycINFO, Embase, and EdSource) using the following search terms:
“competency-based medical education” OR “CBME” OR “competency-based education” OR “competence,” OR “CanMEDS” OR “the Institute of Medicine competencies” OR “ACGME competencies” OR “outcomes project” OR “ACGME”; AND “continuing professional development” OR “CPD” OR “continuing medical education” OR “CME” OR “education, medical, continuing” OR “continuing education”; AND “residency” OR “residency program” OR “residency programme” OR “residency training” OR “postgraduate medical education” OR “PGME” OR “postgraduate training” OR “graduate medical education” OR “GME” OR “education, medical, graduate” OR “internship and residency” OR “medical residency”; AND “family medicine” OR “family practice” OR “general practice” OR “family physician(s)” OR “general practitioner(s)” OR “primary care physician(s)”; AND Canada OR Canadian OR CanMEDS OR United States OR American OR North America.
We selected the keywords based on a review of published literature and several consultations with the librarian, and this process yielded 470 records (after removing duplicates). The search strategy included manual review of key journals, bibliographies of identified articles and websites of relevant organizations (see the full search strategy in Appendix 1 at https://journals.stfm.org/media/3007/hendry-appendices-1-3.pdf). We included all original research, review articles, editorials/commentaries, and regulatory papers.
Stage 4: Study Selection
The inclusion criteria included articles that
- were focused exclusively on family medicine residency or CPD programs;
- discussed CBME;
- were American or Canadian in origin;
- were published in English;
- were available in a full-text version;
- were published between January 1, 2000 (as the ACGME Outcome Project started in 1999) and April 30, 2017.
The article selection process was guided by the approach advocated by Reeves et al.43 We also drew on our team’s extensive experience in scoping reviews.34-39
Step 1: Articles Screened by Title and Abstract. One member of the project team (N.D.), in consultation with the principal investigator (S.K.) screened 470 titles and abstracts based on inclusion criteria. In case of ambiguities, a full-text review (N.D. and S.K.) was conducted using the same inclusion criteria. The decision to include or exclude an article was recorded to ensure procedural rigor.
Step 2: Full-text Review. After completion of Step 1, 114 articles underwent full-text review. N.D. read all articles and S.K. checked 15% of the articles. The project team then discussed any discrepancies, and disagreements were resolved by consensus. Of the 114 articles, we excluded 77 for the following reasons:
- No evidence of the CBME concept (n=13);
- No relevance to CBME implementation practices (n=28);
- Irrelevant to family medicine (n=20);
- Assessment tools description (n=8)
- Undergraduate education (n=5);
- Non-Canadian or US study (n=3).
Thirty-seven articles were included in the final data set (Figure 1).
Criterion #4 (excluding assessment tools description) was selected given the large corpus of research literature covering CBME. As previously reported, one of the important challenges scoping review teams face is the heterogeneity and the expansive volume of the existing literature included in review, which can prohibit effective and meaningful synthesis or quality assessment. In turn, this creates difficulties in performing in-depth analysis and/or information synthesis.40-42
Stage 5: Charting the Data
We completed data extraction using a standardized data extraction form using Microsoft Excel. To ensure rigor of the process and credibility of the findings, we followed Reeves et al’s protocol43 for two or more independent reviewers (C.C., P.H., D.D.) with quality checks from a third party (N.D., S.K.). A detailed description of this step is provided in Appendix 1. We revised the coding manual and specific coding decisions during the coding process based on input from the expert group (C.C., D.D., P.H.; coding manual is provided in Appendix 2 at https://journals.stfm.org/media/3007/hendry-appendices-1-3.pdf).
The final data charting form consisted of two steps. The first step involved gathering article metadata, such as year of publication, title, country where the research was conducted, publication type, research paradigm, and target population. The second step included data relevant to three coding categories: (1) foundation for implementation activity (defined by study authors as documents that serve to support teachers and other educators as they implement the competency-based curriculum framework, including outcomes and standards), (2) implementation activity (defined by Fixsen et al,46 as “a specified set of activities designed to put into practice an activity or program of known dimensions”), and (3) level of implementation practices (defined as “target” or “action target” of interventions).47,48 Using this data extraction form, each expert independently extracted the relevant data from assigned articles in accordance with the criteria defined in Stages 4 and 5. Any ambiguous item was reviewed by the principal investigator (S.K.), who made a final decision. The results were checked by the project team (S.K. and N.D.) for coding consistency. Any changes suggested by the project team were discussed with the experts using a consensus process.
Stage 6: Collating, Summarizing and Reporting Results
Step 1: Analyzing the Data. We synthesized the data according to themes and shared with the expert panel for their feedback.34 Data analysis involved mixed methods, such as quantitative frequency analysis and qualitative content analysis.
Step 2: Reporting Results. We summarized findings in tabular and narrative forms. In order to preserve clarity of reporting strategy, we applied a consistent approach to reporting the findings.32,42,44 This implies that after consultation with the expert panel, we developed a template and applied it to each level of learning continuum (family medicine residency training and CPD). The template included a table summarizing the objectives previously listed in the initial sections of this protocol.
Step 3: Transferability. The final step of the review framework focused on transferability,35 including the applicability of the findings to other similar contexts and the relevance of the findings to knowledge, policy, practice, and research.45
Search and Selection of Scoping Reviews
The original search conducted in April 2017 yielded 891 potentially relevant citations. After screening for duplication and relevance, 114 citations met eligibility criteria based on title and abstract, and the corresponding full-text articles were procured for review. Following full-text review, 37 articles were retained for analysis. Figure 1 represents the flow of the articles from initial identification to final inclusion.
Table 1 reports the general characteristics of the articles included in this study. All included papers (n=37) were published between January 2003 and April 2017, with 76% (n=28) published after 2010 (Figure 2). Almost three-quarters (25/37; 68%) of included studies originated from Canada. Research articles (12/37; 32%), commentary/reflective papers (11/37; 30%), and regulatory (10/37; 27%) comprised most documents included in the review. The number of review articles (4/37; 11%) was underrepresented and there were no editorial opinion articles in the data set. Among 16 studies eligible for classification by a type of research paradigm, 75% (12/16) used a qualitative research approach, 19% (3/16) used a quantitative approach, and one article used mixed methods (1/16; 6%).
Most included studies (27/37; 73%) were identified in published literature, the remaining studies (10/37; 27%) were found in the gray literature, such as websites of CFPC (6/37; 16%), ACGME (2/37; 5%), and American Academy of Family Physicians (2/37’ 5%). The scholarly articles (n=27) were published in 11 different journals with 15 (57%) papers appearing in Canadian Family Physician (10/27; 37%) and Family Medicine (5/27; 19%). Approximately 81% (30/37) of all studies reported residents as their target population, followed by faculty (5/37; 14%) and family physicians (2/37; 5%).
Frequency of Coding Categories: Foundation for Implementation Activity, Implementation Activity, and Level of Implementation Practice
The majority of studies (32/37; 86%) included data coded under the category “foundation for implementation activity” (Table 2). Some of those studies (18/37; 49%) focused solely on “foundation for implementation activity,” but 14 papers (38%) discussed both “foundation for implementation activity” and “implementation activity.” Only a small proportion of the studies (5/37; 13%) concentrated exclusively on the coding category “implementation activity.”
The most commonly described type of foundations for implementation activities was “designing competency-based curriculum” (18/37; 48%), followed by “describing how evaluation/review of competency-based educational program has to be done” (4/37; 11%) and “reviewing competency-based curriculum” (4/37; 11%). The least-used foundation elements were “defining competency-based curriculum outcomes that reflect CanMEDS-FM” (3/37; 8%) and “developing competency-based evaluation objectives” (3/37; 8%).
The coding category “implementation activities” was discussed in 19 papers and fell under two major groups. Almost two-thirds of the studies (11/19; 58%) were assigned to the faculty development (FD) theme with the remaining 42% (8/19) allocated to the curriculum-related activities. The most commonly described FD activity was “training faculty about CBME” (6/19; 33%). Among the curriculum-related activities, the most frequent was “curriculum partial adaptation/implementation” (7/19; 37%) and the least prevalent was “program evaluation” (1/19; 5%).
Over half (20/37; 54%) of the implementation practices were delivered at the program level, followed by institutional/organization level (16/37; 43%). In contrast, only one study (3%) described multiple levels of intervention with individual level of implementation practices being absent (Table 2).
Similarities and Differences in the CBME Implementation Practices Between Family Medicine Residency and CPD
Table 2 highlights the frequency of the three coding categories (foundation for implementation activity, implementation activity, and level of implementation practice) between residency and CPD articles. CBME foundation elements were discussed almost twice as much in residency as in CPD papers, 93% and 57%, respectively. While residency articles focused considerably less (12/30; 40%) on implementation activities, the entire set of CPD studies focused more on FD activities (5/7; 71%) than curriculum activities (2/7; 29%). In contrast, implementation activities in residency papers were equally divided between the two themes: FD and curriculum-related activities. The program level of implementation practices was used more often in residency (15/30; 50%) and CPD (5/7; 71%) studies, followed by institutional/organization level (14/30; 47% and 2/7; 49%, respectively).
The purpose of this study was to identify the range, extent, and nature of CBME implementation practices in family medicine residency and CPD. The major findings of this scoping review were that the vast majority (86%) of included articles concentrated on foundational elements related to designing competency-based curriculum and various implementation strategies, rather than program evaluation or other outcome measures. One-fifth of the articles related to CPD focused on the role for faculty within residency programs rather than the broader issues of how competency frameworks will guide, support, or enable learning and practice improvement initiatives. In addition, there was an exclusive focus on implementation at the program and/or institution/organization levels without a single article focused on the impact of CBME on the individual level.
The review identified three themes: (1) a focus on foundational elements, (2) underevaluated implementation activities, and (3) CBME in CPD as a nascent field.
A Focus on Foundational Elements
After ACGME’s and CFPC’s decisions concerning the strategic initiatives for residency education, scholarship in CBME in family medicine saw an increase. Twenty-eight (76%) articles included in this scoping review were published after 2010. As a complex educational intervention, CBME in residency programs is in its relative infancy. Therefore, our findings that implementation practices concentrated more on foundational elements (Table 2) was anticipated. However, considering the inclusion of milestones (observable markers of an individual’s ability along a developmental continuum) within the CanMEDS 2015 framework,49 and the introduction of entrustable professional activities (EPAs) to guide assessment, it was surprising there were only three articles on defining competency-based curriculum outcomes that reflect CanMEDS-FM.
Underevaluated Implementation Activities
Nineteen studies (51%) discussed specific activities to implement CBME studies divided between residency (n=12) and CPD articles (n=7). Given that implementation of CBME in residency training is recent and evolving, implementation activities mainly focused on FD educational strategies (11/19; 58%). These included the concepts, principles, or intended outcomes for transitioning to CBME (6/19; 33%) with only a limited focus on training curriculum planners about CBME, designing educational meetings, or interprofessional/interdisciplinary education training activities (n=1 each). There were no articles on training initiatives to teach faculty new approaches to clinical supervision of residents within CBME. From a curricular change perspective, seven articles (37%) focused on how CBME could be adapted within traditional clinical rotations.
Another interesting finding was the paucity of formal evaluation studies with only one article focused on approaches to program evaluation. Given that CBME is a complex medical education innovation in the early stages of implementation, the absence of articles to assess the “fidelity of implementation,” defined as the “proportion of program components that were implemented,”50 did not come as a surprise. It has been shown previously that evaluating the fidelity of design elements in CBME can be hindered by the lack of a common framework of what constitutes a CBME program.51,52 Additionally, we identified that the current status of CBME implementation was exclusively focused at the program or institution/organization, rather than individual level. This finding is in parallel with the overall lack of evaluation studies (eg, the impact or outcomes of CBME on individuals) in family medicine residency education and CPD.53
CBME in CPD as a Nascent Field
Finally, there were only seven (19%) articles related to CBME for CPD. Most of these articles focused on FD activities, such as preparing faculty to teach or assess residents in new or different ways. Although this focus is clearly relevant to CPD, there was essentially no scholarship on broader questions of the applicability or relevance of CBME to physicians in practice. This finding corresponds to the previous studies by Lockyer et al,54 Nousiainen et al,55 and Kitto et al53 reporting an overall lack of studies on CBME implementation practices within CPD.
Overall, there was a lack of pertinent or transferable knowledge regarding the applicability of CBME within CPD. The current focus of scholarship in CBME for residency training provided limited insights into how competences or a competency framework will guide physician learning and practice improvement initiatives, or contribute to career development, or what infrastructure will be required to sustain a competency-based approach once residents enter practice.
Strengths and Limitations
To our knowledge, this is the first study to examine competency-based implementation activities in family medicine, an academic discipline known for advancing CBME implementation. The strengths of this scoping review included the assembly of content and methodological expertise from diverse backgrounds.41,42,56 The article selection process (eg, our inclusion/exclusion criteria) followed established methods, integrating the experience of key content expert team members. We acknowledge several limitations of this review. Although we sought to search multiple databases including the gray literature, we limited the scope of the search to articles in English published in Canada and the United States, which have similar residency programs in family medicine.57 Given the exclusive focus on family medicine, the applicability or transferability to other disciplines may be limited.
This study should be a call to action for all those involved in the development and implementation of CBME and the advancement of health professions education. The pattern of implementation activities found in this scoping review reflected a limited focus on a broad range of issues related to fidelity of implementation—that is, whether CBME is being implemented as intended.49 Perhaps the most alarming finding was a lack of studies on the applicability or relevance of CBME to physicians in practice. With the recent drive to create and implement CBME in CPD to improve the quality of care and patient safety, this lack of evidence could seriously hinder implementation practices well into the future.
Drawing on this scoping review and the knowledge gaps identified, we advise that the paucity of literature assessing the application of CBME to practicing family physicians requires our close attention now. As one avenue to explore, we recommend a closer collaboration among researchers in the areas of continuing education, knowledge translation, patient safety, and quality improvement, as well as within stakeholder organizations responsible for each level of medical training and practice. Integration of efforts across all participants should help to develop successful interdisciplinary interventions to improve professional performance and patient outcomes.53 Before the wave of implementation activities truly commences in earnest across Canada and the United States, there are still many questions that need to be answered, including:
- How will competencies guide physician learning or practice improvement within their scope of practice?
- What outcomes or outcome measures can competency-based medical education enable physicians to pursue, either individually or collectively?
- How can competency-based CPD enable or enhance interprofessional CPD?
We hope this collated resource will help educators become more aware of the existing menu of implementation practices and the gaps in family medicine residency and CPD. There is a need for ongoing and rigorous evidence-gathering to enable the implementation of CBME in CPD.
- Frank JR, Snell LS, Cate OT, et al. Competency-based medical education: theory to practice. Med Teach. 2010a;32(8):638-645. https://doi.org/10.3109/0142159X.2010.501190
- Campbell C, Silver I, Sherbino J, Cate OT, Holmboe ES. Competency-based continuing professional development. Med Teach. 2010;32(8):657-662. https://doi.org/10.3109/0142159X.2010.500708
- Kruse J. Social accountability across the continuum of medical education: a call for common missions for professional, accreditation, certification, and licensure organizations. Fam Med. 2013;45(3):208-211.
- Swing SR. The ACGME outcome project: retrospective and prospective. Med Teach. 2007;29(7):648-654. https://doi.org/10.1080/01421590701392903
- Whitehead CR, Martin D, Fernandez N, Younker M, Kouz R, Frank J, Boucher A. Integration of CanMEDS expectations and outcomes. Members of the FMEC PG consortium. 2011. https://www.afmc.ca/pdf/fmec/15_Whitehead_CanMEDS.pdf. Assessed September 17, 2018.
- Sherbino J, Kulasegaram K, Worster A, Norman GR. The reliability of encounter cards to assess the CanMEDS roles. Adv Health Sci Educ Theory Pract. 2013;18(5):987-996. https://doi.org/10.1007/s10459-012-9440-6
- Hawkins RE, Welcher CM, Holmboe ES, et al. Implementation of competency-based medical education: are we addressing the concerns and challenges? Med Educ. 2015;49(11):1086-1102. https://doi.org/10.1111/medu.12831
- Eva KW, Bordage G, Campbell C, et al. Towards a program of assessment for health professionals: from training into practice. Adv Health Sci Educ Theory Pract. 2016;21(4):897-913. https://doi.org/10.1007/s10459-015-9653-6
- Tannenbaum D, Kerr J, Konkin J, et al. Triple C competency-based curriculum. Report of the Working Group on postgraduate curriculum review – Part 1. Mississauga, ON: College of Family Physicians of Canada; 2011, www.cfpc.ca/uploadedFiles/Education/_PDFs/WGCR_TripleC_Report_English_Final_18Mar11.pdf. Accessed August 24, 2018.
- Allen T, Brailovsky C, Rainsberry P, et al. Defining competency-based evaluation objectives in family medicine: dimensions of competence and priority topics for assessment. Can Fam Physician. 2011;57(9):e331-e340.
- Oandasan I, Saucier D, eds. Triple C Competency-based curriculum report – Part 2: Advancing implementation. Mississauga, ON: College of Family Physicians of Canada; 2013, www.cfpc.ca/uploadedFiles/Education/_PDFs/TripleC_Report_pt2.pdf. Accessed September 27, 2018.
- Iglar K, Whitehead C, Takahashi SG. Competency-based education in family medicine. Med Teach. 2013;35(2):115-119. https://doi.org/10.3109/0142159X.2012.733837
- Lacasse M, Théorêt J, Tessier S, Arsenault L. Expectations of clinical teachers and faculty regarding development of the CanMEDS-Family Medicine competencies: laval developmental benchmarks scale for family medicine residency training. Teach Learn Med. 2014;26(3):244-251. https://doi.org/10.1080/10401334.2014.914943
- McEwen LA, Griffiths J, Schultz K. Developing and successfully implementing a competency-based portfolio assessment system in a postgraduate family medicine residency program. Acad Med. 2015;90(11):1515-1526. https://doi.org/10.1097/ACM.0000000000000754
- Schultz K, Griffiths J. Implementing competency-based medical education in a postgraduate family medicine residency training program: a stepwise approach, facilitating factors, and processes or steps that would have been helpful. Acad Med. 2016;91(5):685-689. https://doi.org/10.1097/ACM.0000000000001066
- Edwards FD, Frey KA. The future of residency education: implementing a competency-based educational model. Fam Med. 2007;39(2):116-125.
- Tudiver F, Rose D, Banks B, Pfortmiller D. Reliability and validity testing of an evidence-based medicine OSCE station. Fam Med. 2009;41(2):89-91.
- Jansen KL, Rosenbaum ME. The state of communication education in family medicine residencies. Fam Med. 2016;48(6):445-451.
- Thomas RE, Kreptul D. Systematic review of evidence-based medicine tests for family physician residents. Fam Med. 2015;47(2):101-117.
- Meili R, Buchman S, Goel R, Woollard R. Social accountability at the macro level: framing the big picture. Can Fam Physician. 2016;62(10):785-788.
- Organek AJ, Tannenbaum D, Kerr J, et al. Redesigning family medicine residency in Canada: the triple C curriculum. Fam Med. 2012;44(2):90-97.
- Saucier D, Shaw E, Kerr J, et al. Competency-based curriculum for family medicine. Can Fam Physician. 2012;58(6):707-708, e359-e361
- Resident Doctors of Canada. Implementing a competency-based approach to medical education. 2016. https://residentdoctors.ca/wp-content/uploads/2016/07/POSITION-PAPER-CBME_EN.pdf. Accessed September 15, 2018.
- FMEC PG. The Future of Medical Education in Canada: Postgraduate Project. Public Report. 2012. https://afmc.ca/pdf/fmec/FMEC_PG_Public_Report_EN.pdf Assessed May 22, 2019.
- Accreditation Council for Graduate Medical Education. Outcome Project: General competencies. 1999. http://www.acgme.org/outcome/comp/compFull.asp. Accessed September 20, 2018.
- Accreditation Council for Graduate Medical Education, American Board of Medical Specialties. Toolbox of assessment methods. 2000. http://njms.rutgers.edu/culweb/medical/documents/ToolboxofAssessmentMethod.pdf. Accessed 2018, September 20.
- Stephens MB, McKenna M, Carrington K. Adult learning models for large-group continuing medical education activities. Fam Med. 2011;43(5):334-337.
- Kitto SC, Bell M, Goldman J, et al. (Mis)perceptions of continuing education: insights from knowledge translation, quality improvement, and patient safety leaders. J Contin Educ Health Prof. 2013;33(2):81-88. https://doi.org/10.1002/chp.21169
- Sargeant J, Bruce D, Campbell CM. Practicing physicians’ needs for assessment and feedback as part of professional development. J Contin Educ Health Prof. 2013;33(1)(suppl 1):S54-S62. https://doi.org/10.1002/chp.21202
- Boucher A, Frank JR, Melle V, Oandasan I, Touchie C. Competency-based medical education in Canada: A white paper commissioned by the AFMC board of directors. 2017. https://mededconference.ca/sites/default/files/AFMCCompetencyBasedMedicalEducation_en.pdf. Accessed August 18, 2018.
- Ellaway RH, Palacios Mackay M, Lee S, et al. The impact of a national competency-based medical education initiative in family medicine. Acad Med. 2018;93(12):1850-1857. https://doi.org/10.1097/ACM.0000000000002387
- Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19-32. https://doi.org/10.1080/1364557032000119616
- Pham MT, Rajić A, Greig JD, Sargeant JM, Papadopoulos A, McEwen SA. A scoping review of scoping reviews: advancing the approach and enhancing the consistency. Res Synth Methods. 2014;5(4):371-385. https://doi.org/10.1002/jrsm.1123
- Grant RE, Sajdlowska J, Van Hoof TJ, Kitto S. Conceptualization and reporting of context in the North American continuing medical education literature: a scoping review protocol. J Contin Educ Health Prof. 2015;35(2)(suppl 2):S70-S74. https://doi.org/10.1097/CEH.0000000000000010
- Olding M, McMillan SE, Reeves S, Schmitt MH, Puntillo K, Kitto S. Patient and family involvement in adult critical and intensive care settings: a scoping review. Health Expect. 2016;19(6):1183-1202. https://doi.org/10.1111/hex.12402
- Horsley T, Lockyer J, Cogo E, Zeiter J, Bursey F, Campbell C. National programmes for validating physician competence and fitness for practice: a scoping review. BMJ Open. 2016;6(4):e010368. https://doi.org/10.1136/bmjopen-2015-010368
- Tricco AC, Ashoor HM, Cardoso R, et al. Sustainability of knowledge translation interventions in healthcare decision-making: a scoping review. Implement Sci. 2016;11(1):55. https://doi.org/10.1186/s13012-016-0421-7
- Jeong D, Presseau J, ElChamaa R, et al. Barriers and facilitators to self-directed learning in continuing professional development: A Scoping review. Acad Med. 2018;93(8):1245-1254. https://doi.org/10.1097/ACM.0000000000002237
- Chappell K, ElChamaa R, Jeong D, et al. Conceptualization and operationalization of certification in the US and Canadian nursing literature. J Nurs Adm. 2018;48(5):238-246. https://doi.org/10.1097/NNA.0000000000000608
- Brien SE, Lorenzetti DL, Lewis S, Kennedy J, Ghali WA. Overview of a formal scoping review on health system report cards. Implement Sci. 2010;5(1):2. https://doi.org/10.1186/1748-5908-5-2
- Levac D, Colquhoun H, O’Brien KK. Scoping studies: advancing the methodology. Implement Sci. 2010;5(1):69-77. https://doi.org/10.1186/1748-5908-5-69
- Thomas A, Lubarsky S, Durning SJ, Young ME. Knowledge syntheses in medical education: demystifying scoping reviews. Acad Med. 2017;92(2):161-166. https://doi.org/10.1097/ACM.0000000000001452
- Reeves S, Barr H, Birch I, et al. BEME Review Protocol. Surrey, UK; 2014:1-18,
- Tricco AC, Lillie E, Zarin W, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467-473. https://doi.org/10.7326/M18-0850
- Kitto SC, Chesters J, Grbich C. Quality in qualitative research. Med J Aust. 2008;188(4):243-246. https://doi.org/10.5694/j.1326-5377.2008.tb01595.x
- Fixsen DL, Naoom SF, Blase KA, Friedman RM, Wallace F. Implementation research: a synthesis of the literature (No. FMHI publication #231). Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, National Implementation Research Network; 2005. https://nirn.fpg.unc.edu/sites/nirn.fpg.unc.edu/files/resources/NIRN-MonographFull-01-2005.pdf. Assessed April 22, 2018.
- Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012;50(3):217-226. https://doi.org/10.1097/MLR.0b013e3182408812
- Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8(1):139-150. https://doi.org/10.1186/1748-5908-8-139
- Frank JR, Snell L, Sherbino J, editors. CanMEDS 2015 Physician Competency Framework. Ottawa: Royal College of Physicians and Surgeons of Canada; 2015. http://www.royalcollege.ca/rcsite/documents/canmeds/canmeds-full-framework-e.pdf. Accessed May 22, 2019.
- Mowbray CT, Holter MC, Teague GB, Bybee D. Fidelity criteria: Development, measurement, and validation. Am J Eval. 2003;24(3):315-340. https://doi.org/10.1177/109821400302400303
- Glasgow NJ, Wells R, Butler J, Gear A. The effectiveness of competency-based education in equipping primary health care workers to manage chronic disease in Australian general practice settings. Med J Aust. 2008;188(S8)(suppl):S92-S96. https://doi.org/10.5694/j.1326-5377.2008.tb01755.x
- Van Melle E, Frank JR, Holmboe ES, Dagnone D, Stockley D, Sherbino J; International Competency-based Medical Education Collaborators. Stockley, Sherbino J, on behalf of the International Competency-based Medical Education Collaborators. A core components framework for evaluating implementation of competency-based medical education programs. Acad Med. 2019;94(7):1002-1009. https://doi.org/10.1097/ACM.0000000000002743
- Kitto S, Danilovich N, Delva D, et al. Uncharted territory: knowledge translation of competency-based continuing professional development in family medicine. Can Fam Physician. 2018;64(4):250-253.
- Lockyer J, Bursey F, Richardson D, Frank JR, Snell L, Campbell C; ICBME Collaborators. Competency-based medical education and continuing professional development: A conceptualization for change. Med Teach. 2017;39(6):617-622. https://doi.org/10.1080/0142159X.2017.1315064
- Nousiainen MT, Caverzagie KJ, Ferguson PC, Frank JR; ICBME Collaborators. Implementing competency-based medical education: what changes in curricular structure and processes are needed? Med Teach. 2017;39(6):594-598. https://doi.org/10.1080/0142159X.2017.1315077
- Daudt HM, van Mossel C, Scott SJ. Enhancing the scoping study methodology: a large, inter-professional team’s experience with Arksey and O’Malley’s framework. BMC Med Res Methodol. 2013;13(1):48. https://doi.org/10.1186/1471-2288-13-48
- O’Neill TR, Peabody MR, Puffer JC. Differences in Canadian and US medical student preparation for family medicine. Fam Med. 2016;48(10):770-774.