Implementation of Competency-Based Medical Education in a Family Medicine Residency

Graduate medical education emphasizes that residents must meet predefined competencies before graduation and board certification. The goal of competency-based medical education (CBME) is to evaluate these competencies in predefined outcomes rather than prespecified metrics. However, the process and the impact of evaluating competency needs to be better defined. Recent literature suggests that successful implementation of CBME involves an assessment system, timely evaluation, and monitoring progress during training.^1-3 A narrative review of CBME methods found a lack of research on competency-based assessments methods and implored programs to evaluate resident perceptions of CBME.¹

During the 2024-2025 academic year, The Ohio State University (OSU) Family Medicine Residency Program implemented CBME through on-demand assessments, structured individualized learning plans (ILPs), and faculty coaching of residents, based on recommendations from the Society of Teachers in Family Medicine (STFM) CMBE Task Force.⁴ Our study aims to assess residents’ perceptions of implemented components of CBME, determine if implementation of CBME resulted in increased competency and knowledge, and evaluate whether on-demand evaluations provided more comprehensive data on resident milestones.

We conducted a single-institution educational program implementation evaluation from July 2024 to June 2025, utilizing both retrospective and prospective data. This study was conducted at OSU and received institutional review board approval (ID 20250296). The study was conducted while the residency participated in the STFM CBME pilot program.⁵

The study sample included 36 family medicine residents at OSU, including 27 active residents and nine residents who graduated in 2024. There were no exclusion criteria for this study. Retrospective data were collected for a year pre-exposure as a baseline. Residents in training during the study period were consented to complete a survey. 

Elements of CBME were implemented in July 2024 including using a direct observation evaluation form created by STFM, requiring completion of quarterly individualized learning plans (ILPs) utilizing a template created by STFM , and having faculty serve as coaches.^4-5 Faculty were encouraged to complete direct observation evaluations frequently, with the goal of each resident receiving one evaluation per week. Faculty also received training on how to serve as a coach and guide learners in developing ILPs. 

Resident perception of CBME was assessed via a one-time postexposure survey using questions with a Likert scale. Data were also collected from the 2024 and 2025 Accreditation Council of Graduate Medical Education (ACGME) surveys on “Satisfaction With Faculty Members Feedback.” We evaluated resident knowledge by collecting in-training exam (ITE) scores from 2023 and 2024. Additionally, all 19 milestones subcompetency scores were compared for each resident stratified by postgraduate year (PGY) from preexposure to postexposure. We assessed the impact of direct observation evaluations by evaluating the total number of evaluations from pre-exposure to postexposure. As the evaluations mapped to milestone subcompetencies, we also assessed how many milestone subcompetencies had data directly populated from the evaluations, comparing pre-exposure to postexposure.

We used descriptive statistics for survey data, number of evaluations completed, and number of milestone subcompetencies with data. Given unequal variances, we used a one-sided Welch’s t test to compare differences of ITE scores and milestone subcompetency scores. 

The resident survey was completed by 18 of the 27 eligible residents (Table 1). On average, residents agreed that implementing CBME was a positive change, and direct observations were beneficial. Residents were neutral on the benefits of ILPs. Accreditation Council for Graduate Medical Education survey data showed an increase in satisfaction with faculty feedback from a 3.7/5.0 pre-exposure to 4.0/5.0 postexposure. ITE scores increased from a mean of 421(SD=65) pre-exposure to 525 (SD=170) postexposure across all PGYs (P<.00005) and for each individual PGY (Table 2). Except for statistically significant increases in SBP 1 and SBP 4 for PGY2s, no other milestone subcompetencies showed a significant change postexposure. The number of total evaluations increased from 646 to 1,173 postexposure, of which 679 evaluations were direct observation evaluations (Table 3). Out of the 19 milestone subcompetencies, a mean of 11.6 had linked data available pre-exposure, and all 19 had data available postexposure (Table 4). 

Overall, residents viewed direct observations positively for evaluation of competency and resulted in an 82% increase in evaluations completed. Although implementing CBME did not directly improve milestone subcompetency scores, it did increase milestone data mapped from evaluations providing invaluable quantitative and qualitative data to the Curriculum Competency Committee. Faculty coaching was viewed positively, but the overall attitude toward performing ILPs was mixed.

Evaluating for competency rather than time-based requirements in medical education is an important shift for training residents transitioning from medical school to independently-practicing physicians.⁴ Survey data from program directors suggest many have graduated residents despite concerns.^6-7 This highlights the importance in the shift toward competency-based, time-variable training in medical education, an approach that has been embraced in higher education in other fields for many years.^8-9 This study highlights how elements of CBME can be used to gather high volumes of data, which can be used to identify gaps in competency and guide learning plans for learners to achieve competency. A study of implementing CBME at Canadian family medicine residencies also found an increase in evaluations, with an average of 150 evaluations per resident over 3 years.¹⁰ Increased assessment data helped these residencies identify red flags and competency trajectories sooner.¹⁰ 

It is important to note increased quantity of evaluations does not correlate with quality, as one study showed implementation of a CBME evaluation system led to variable assessment quality, distracted from learning, and was associated with resident anxiety.¹¹     

Limitations of this study include a small study sample and short duration of study. Although there was a large increase in the quantity of evaluations, most evaluations were completed by a few faculty, often at times when reminders were sent. Therefore, although exposure led to increased evaluations and coaching, the workload for faculty may not be sustainable. Future research and policy should focus on improved workflows and faculty protected time to allow for implementation of CBME. Although ITE scores improved with our exposure, the ITE was completed 3 months after implementation. Therefore, confounding factors such as class differences and didactic changes may better explain improvements in ITE scores.   

The results of this 1-year project indicate a positive trajectory for implementation of CBME in that it leads to more frequent assessment and guidance for learners. Further investigation should occur to assess long-term impact of CBME on practice outcomes. Furthermore, validation studies should occur for current direct observation assessment forms, to ensure they capture appropriate stages towards competency.