Assessment of Agenda Setting With ACGME Milestones in Family Medicine Residents

Medical educators are responsible for providing residents with timely, constructive feedback. The Accreditation Council for Graduate Medical Education (ACGME) Family Medicine Milestones are assessments of competencies organized around six core domains: patient care, medical knowledge, systems-based practice, practice-based learning and improvement, professionalism, and communication.¹ Milestone ratings have been found to be a “viable multidimensional tool” to measure resident competence and progression.²

Agenda setting involves setting expectations for the visit and organizing a comprehensive list of topics to be discussed between patient and provider, thereby reducing the number of unaddressed health concerns.³ A provider’s ability to proactively negotiate a visit agenda is a skill shown to improve deficiencies in communication, but is not always taught as part of standard medical education.⁴ Therefore, communication is a specific area of focus for our resident direct observations (DO) of patient encounters.

DO increases the frequency of feedback, helps identify clinical deficiencies, increases resident confidence, and improves resident communication skills.⁵ In 2015, faculty in the Department of Family and Community Medicine began to assess residents with DO forms focused on aligning the 2014 Milestones¹ with portions of a clinical encounter. By targeting communication skills via agenda setting earlier in residency, faculty aimed to build resident confidence during patient interactions. DO forms are one of the many components considered in creating a Milestone score, and the impact of DO forms on Milestones scores has not been previously examined.

This study examined the relationship between ACGME communication milestone achievement and resident ability to set a visit agenda, as measured by DO forms.

This study was approved by the Penn State College of Medicine Institutional Review Board.

Biannual (fall, spring) ACGME Milestone scores from all residents (n=56 residents) from July 2015-June 2020 were gathered as one of the data sources assessed based on the 2014 ACGME Family Medicine Milestones Table 1.¹

DOs occurred in two Mid-Atlantic suburban clinics in an opposed residency program that is part of a large academic medical center. Information collected from the observation forms included assessment of behaviors in the following skill categories: medical interview, physical exam, assessment and plan, counseling skills/shared decision making, relationship skills, and organization and efficiency. Some categories contained subcategories. Faculty observed the entire encounter and circled, highlighted, or bolded any resident behaviors witnessed and struck through behaviors that could have been, but were not performed (Supplementary Figure 1).

Open-ended fields for additional comments were included in each section and at the end for overall goals. Faculty completed the electronic form immediately following the patient encounter, reviewed feedback with the resident prior to the end of the clinic session, and a copy of the form was provided to the resident and their advisor. The forms were uploaded into New Innovations management software and used as part of assessment review prior to and during each Clinical Competency Committee (CCC). The CCC evaluations are completed through committee (ie, residency faculty) consensus through review of multiple data sources (eg, In-Training Exam scores, numbers of procedures completed and logged, etc), including DO forms.

We gave a rating to each resident as a percentage of the identified items. We calculated total ACGME scores to assess potential differences in overall communication, individual levels, and general performance. We used Spearman and Pearson correlations and two-sample paired t tests to analyze results by postgraduate year (PGY). We used Bonferroni corrections for multiple comparisons (n=24). We entered data into REDCap software⁶ and analyzed using the R statistical program version 4.0.2 (R Foundation for Statistical Computing, Vienna, Austria).

We analyzed a total of 246 ACGME CCC evaluations and 215 DO forms from a 5-year period. Not all DO forms completed during this time period were submitted into the New Innovations system, making them inaccessible for this review. There was a significant, positive association between agenda setting and the total Milestone score (r[190]=.15, P=.034) when examining scores from the fall time frame (July 1-December 31, 2015-2020). In spring (January 1-June 30, 2015-2020), we identified significant positive associations between agenda-setting and the communication milestone, C-1 [r(190]=.17, P=.020), as well as total communication milestone scores (r[186]=.16, P=.031). These significant associations are only trends following adjustment for Bonferroni corrections (α now <.002). Overall, significant progression was seen for consecutive years in both communication milestones (t=-15.06, P<.001) and agenda-setting (t=-12.26, P<.001). A summary of all significant findings can be found in Table 2.

Our study found agenda setting to be significantly associated with both total communication and total Milestone scores for PGY1s. Success in agenda-setting in the fall was associated with an increase in PGY1 total Milestone scores. We expected PGY1 improvement in all Milestones in the spring, due to the longer training period, but this was only true for the communication Milestone. Communication is a critical area for faculty to focus early in PGY1 training, to build confidence in resident-patient interactions, which are reflected in CCC assessments.⁵

Agenda setting is a foundational competency impacting PGY1s. It may be that PGY1 Milestone scores on communication were more affected by agenda setting, when compared to PGY2s and PGY3s, because more senior residents have other communication factors contributing to and buffering their scores. Moreover, PGY1s who set an agenda appear to have greater success in communicating with their patients, as both resident and patient are more likely to be satisfied with topics covered during the encounter if topics were mutually agreed upon at the start.³

Our study is limited by missing DO forms, use of correlations, and the potential for unknown confounders to affect the results of the study. Additionally, this study assessed resident Milestone scores in one residency program in Central Pennsylvania, limiting generalizability. Furthermore, there may be a recall bias due to latency in completion of the DO forms, as they were completed the end of the patient encounter. There is also a need for a follow-up study to analyze interrater reliability of agenda setting assessment. A more recent version of the ACGME Milestones was released in 2019,^{7, 8} where agenda setting and communication Milestones were fundamentally the same, however, in the new version these were assigned to a level 2 versus a level 3 assessment. These data may not reflect the changes associated with the newest version.