Resident Self-assessment and Clinical Competency Committee Evaluations in Family Medicine: A Single-Institution Study

One of the critical tasks for family medicine residency programs is to provide trainees feedback that measures their performance against standards that reflect the wide domain of competencies in which proficient family physicians routinely function. In 2014, leaders from family medicine organizations across the United States and the Accreditation Council for Graduate Medical Education (ACGME) developed milestones with the goal of providing “meaningful, formative feedback to residents on their progress towards competence in the specialty,” ¹ with emphasis on feedback that is specific, outcome focused, and work based. ² Family medicine milestones include six domains, with 19 individual milestones across these domains. ³ Clinical Competency Committees (CCC), composed of faculty, are tasked with overseeing milestone assessments at each US graduate medical education (GME) program. ⁴

Challenges accompanying milestone assessments include developing residents’ understanding of milestones and structuring assessments in a manner that encourages self-reflection and growth. ⁵ Self-assessment has the potential to address these challenges ⁶ and is emerging as an important tool in medical education because of its potential to align learner/instructor expectations, ^{7, 8} catalyze self-reflection, ⁸ identify program deficiencies and guide curriculum development, ⁹ and encourage lifelong habits of accurate self-assessment driving self-directed learning and improvement. ^{7, 10, 11}

Several non-primary care oriented residency programs have introduced self-assessment into milestone assessments ^{12, 13}, ^14-23; however, we have not seen milestone self-assessment introduced in a family medicine residency program in the US literature. This article shares the development of a self-assessment process for family medicine residents and the findings of a comparison between CCC assessments and resident self-assessments.

The University of Texas Medical Branch (UTMB) Family Medicine Residency Program is a 10-10-10 ACGME-accredited program with 30 residents located in Galveston, Texas. In fall 2020, leadership incorporated resident self-assessment into the milestone assessment process in response to the ACGME annual resident survey, ²⁴ which identified resident concerns about the quality of faculty feedback. On further exploration, residents expressed a desire for an increased role in assessing their performance. The UTMB Institutional Review Board deemed this study exempt from review.

Each postgraduate year (PGY) underwent milestone assessments twice per year. Prior to self-assessments, residents attended a didactics lecture and received a copy of the ACGME Family Medicine Milestones 2.0, where they received guidance on self-assessment and descriptions of each milestone (see supplemental material). Residents rated themselves from level 1 (novice) to level 5 (expert), with scores between levels (eg, 3.5) reflecting demonstration of some, but not all activities described at the higher level. ³ Residents were given 2 weeks to complete self-assessments.

Once received, the residency staff compiled data from resident self-assessments (Figure 1). The CCC then met and discussed every resident’s observed performance of specific practices or skills described for each milestone and adjusted their final scores if a resident’s self-score was inconsistent with faculty observations. Final scores were determined by consensus.

Residents and faculty advisors were provided feedback on reasons for any differences between self-assessment and final scores 1 to 2 weeks following the CCC meeting. They also received feedback on self-assessment patterns for residents who consistently scored themselves lower or higher than CCC assessment. Faculty advisors and residents then met to discuss CCC feedback. Residents were invited to share any milestone activities that may not have been observed by faculty (eg, active involvement in advocacy).

We analyzed self-assessment and CCC final milestone scores from 30 residents in fall 2020 and spring 2021. Mean and standard deviation of average milestone score across 19 individual milestones by self-assessment and CCC assessment at each term were reported. We used box plot to demonstrate the distribution of scores stratified by PGY. We conducted repeated measure analysis of variance to examine within-subject effects (assessor: self-assessment vs CCC; term: fall 2020 vs spring 2021) and between-subject effects (gender and PGY) on the outcome of milestone score. We performed all analyses with SAS software version 9.4 (SAS Institute, Cary, NC).

 A total of 60 self-assessments and 60 CCC assessments were conducted. In the study period, the CCC score was similar to self-assessment. There were larger variations in resident self-assessment scores than in CCC scores. Both CCC and self-assessment scores increased by PGY; however, scores were not different between fall 2020 and spring 2021 (Figure 2).

Results from multivariable analyses showed a significant three-way interaction between assessor, terms, and PGY (P<.0001), under which all of the two-way interactions (assessor and terms, assessor and PGY, and terms and PGY) were significant. CCC assessment scores increased from fall 2020 to spring 2021 (P=.0009), while resident self-assessment scores did not. As compared to resident self-assessment, CCC assessment also revealed a larger difference between PGY. CCC assessment indicated a larger difference between PGY in spring 2021 than in fall 2020 (Table 1 ).

Resident self-assessment provides the CCC insight when a resident’s self-perception of performance may not accurately reflect observed behaviors or skills, highlighting specific areas for growth that the resident may not otherwise recognize. It also provides an opportunity to address a resident’s underestimation of performance, decreasing risk for imposter syndrome. ²⁵

There were several limitations to this study. First, we studied a small cohort of residents for a short duration from a single residency program. We grouped assessments without analyzing results for the 19 specific milestones or for the six milestone domains. Finally, the CCC used resident self-assessments as a starting point for discussions about final milestone scores. An independently-performed CCC assessment would have reduced the potential for bias in CCC final scores; however, independent assessments would have failed to honor residents’ requests to participate in assessments.

Areas for future study include multisite studies that follow large numbers of residents across all years of training. Data gathered from milestone self-assessments could also be compared to national milestones reports published by ACGME, across specialties or family medicine departments at other institutions, or to milestone approaches in other countries.

Milestone assessments can often be relayed as one-way data that frustrates or discourages learners. However, our milestone process fosters a two-way conversation about performance rooted in self-reflection. ²⁶ Self-assessments using milestones can help direct improvement in the many domains and skills required of a competent family physician.

The authors thank Christen Walcher for editing the manuscript.