Comparing Evaluations of Family Medicine Residents by Gender and Ethnicity

Ensuring accurate, objective, and unbiased evaluation and assessment of resident trainees is a critical element within graduate medical education programs. These evaluations are often conducted in relationship to the standards set forth in the Accreditation Council for Graduate Medical Education (ACGME) milestones for each training program.¹ Because resident evaluation and assessment data are critical to determine promotion, evaluate clinical competence, structure remediation, and determine whether resident physicians have met graduation targets, ensuring accuracy of assessments free from biases is paramount. To date, however, little is known about how trainee demographics impact evaluation within family medicine and whether biases of evaluation are present in this cohort.

In other specialties, qualitative and quantitative differences in evaluations have been well-described. Anesthesia residents who identify as women, for example, are less likely to be described with agentic qualities such as independence, competence, and efficacy early in their training when compared to men counterparts.² Similarly, literature has described differences in quantitative and qualitative evaluations of emergency medicine residents, highlighting that women are less likely to receive consistent feedback when struggling and are more likely to receive negative feedback on traditionally masculine traits such as assertiveness and autonomy.^3,4 Quantitatively, literature from general surgery training programs has demonstrated lower milestone assessments across nearly all subcompetencies among women residents, although both men and women residents achieved scores consistent with competence for transition to independent practice prior to graduating residency.⁵ In internal medicine training, numeric ratings of women residents have demonstrated a peak-and-plateau pattern after postgraduate year (PGY) 2, whereas men residents continued to show quantitative evaluation improvements.⁶ Similar findings have been noted based on self-identified race, with data suggesting that residents identifying as ethnic minorities and women receive lower quantitative ratings on ACGME milestone domains when compared to their White men counterparts.^7,8

The influence of residency evaluations on the postgraduation careers of physicians cannot be understated. Literature has suggested that residents who have action taken for performance concerns have lower graduation rates than peers.⁹ If discrepancies exist among residents based on gender or ethnicity, they could have a substantial impact on the career trajectories of residents from ethnic minority backgrounds. While prior work has focused on other specialties, little is known whether similar trends exist in family medicine residency training.^2,3,6,10 This possible trend is particularly important in family medicine, where a plurality of physicians in residency are women.¹¹ Prior work has identified that women residents may be more likely to receive more feedback on communication skills than men counterparts, often in relation to opportunities for improvement for the resident.⁵ With the emphasis on continuity relationships and outpatient visits in family medicine, women residents may be disproportionately impacted if this trend occurs in family medicine.

To better understand whether similar differences exist in family medicine residency training, we evaluated demographic differences in quantitative and qualitative evaluations of residents based on identified gender and ethnicity. We hypothesized that quantitative milestone ratings and qualitative language use—including agentic, communal, feminine, and masculine language—would differ based on resident gender and ethnicity, consistent with findings from other specialties. We explored whether other linguistic features, including hedging language, differed across demographic groups.

We conducted a retrospective observational study of milestone evaluations from a single urban academic family medicine residency program. Data included all submitted evaluations and milestone-level assessments completed between June 2020 and November 2024. Resident evaluations are submitted and stored via a centralized residency management system (MedHub). During the study period, the residency program increased its size from 27 to 36 residents across three postgraduate years. Evaluations were completed by supervising faculty from family medicine and other specialties participating in required residency rotations, including inpatient, outpatient, and obstetrics experiences, as well as following outpatient clinic precepting sessions. These evaluations were compiled biannually at clinical competency committee (CCC) meetings where consensus ACGME milestone evaluations were generated. The unit of analysis was the resident for each academic year, with residents contributing repeated observations across multiple academic years as applicable. Resident demographic variables included self-described gender and ethnic status; descriptive statistics summarized resident demographics, distribution across academic years and PGY levels, and evaluation counts. The demographic makeup of the residents is outlined in Table 1.

For analyzing quantitative differences in milestone ratings, the primary outcomes were milestone scores for six core competencies: interpersonal and communication skills (ICS), medical knowledge (MK), practice-based learning and improvement (PBLI), patient care (PC), professionalism (PROF), and systems-based practice (SBP). Milestone scores were treated as continuous variables. Covariates were selected a priori based on conceptual relevance to milestone assessment: PGY (modeled as a continuous variable to represent developmental progression), gender, ethnicity, evaluation exposure (defined as the number of evaluations completed within the same academic year), and academic year (included as a categorical fixed effect to account for cohort-level and system-level differences across calendar years). To account for differences in residents’ duration in the study, we used the Kruskal-Wallis test to compare the mean annual number of evaluations received per resident across ethnic groups.

We conducted unadjusted comparisons of milestone scores by gender and ethnicity using two-sample t tests, both pooled across years and stratified by academic year and PGY. Primary analyses used linear mixed-effects regression models fit separately for each competency. Models included fixed effects for PGY, gender, ethnicity, evaluation exposure within the academic year, and academic year, with a random intercept for resident to account for repeated observations. Likelihood ratio tests compared mixed-effects models to corresponding linear models without random effects.

We used LIWC-22 (linguistic inquiry and word count; Pennebaker Conglomerates Inc) to analyze narrative comments for each submitted evaluation during the study period.^12,13 This tool allows for quantitative analysis of written language—providing metrics like word count while also quantifying the presence of specific words within previously developed dictionaries. Based on prior literature analyzing gender differences in narrative evaluation using LIWC-22,¹⁴ we assessed for differences in the use of “masculine” terms, “feminine” terms, “agentic” terms (terms that connotate an individual’s ability), and “communal” terms (terms that highlight an individual’s desire to relate and cooperate) using pre-developed LIWC-22 dictionaries.^15-18 Using the LIWC-22 score for each domain, we evaluated summary statistics of the domains by resident gender, resident ethnicity, and PGY.

In addition to LIWC-22 analysis, we analyzed word use through n-gram analysis,¹⁹ a natural language processing method that converts free-text comments into discrete words and phrases. N-gram analysis allows creation of a dataset indicating the presence or absence of single words (unigrams, eg, “clinician” or “brilliant”). Using previously published research on hedging language in evaluation,²⁰ we identified and categorized the unigrams into hedging language to explore potential differences by resident gender and ethnicity (Table 2). Using this category as a binary variable (present vs absent hedging language), we evaluated summary statistics of the hedging language by resident gender, ethnicity, and PGY. Similar to work that has been done in other settings, we evaluated specific terms of confidence (including words such as “confident,” “confidence,” or “confidently”).²¹

We then performed mixed-effects linear regression analyses to assess the association of the LIWC-22 categories (agentic, communal, masculine, feminine, word count) with resident gender and ethnicity, adjusted for resident PGY, clustered on rater (random effects). We also performed mixed-effects linear regression analyses to assess the association of the hedging language with resident gender and ethnicity, adjusted for PGY, clustered on rater (random effects). For all analyses, statistical significance was determined using a P value of .008 (using a Bonferroni correction for multiple comparisons). All statistical analyses and additional text analysis were completed using Stata 19.5 (StataCorp) and LIWC-22.

This project was reviewed and determined to qualify as quality improvement by the University of Pennsylvania’s Institutional Review Board.

The dataset included 152 resident–academic year observations from 73 residents (79% women) and 3,184 total evaluations completed by 67 unique faculty. Residents received a mean of 16.6 (SD 6.6) faculty evaluations during the study period, with the number varying by academic year (Table 1). Of the rotation evaluations, 74% were in reference to women and 34% were of residents classified as ethnic minorities (Black, Asian, and/or Hispanic). The mean number of evaluations received per resident per academic year did not differ significantly by ethnicity (Kruskal-Wallis P = 0.1). Residents were evenly distributed across PGY-1 through PGY-3, with few PGY-4 observations. The evaluations of the cohort were predominantly of Caucasian individuals (36% total evaluations, n = 1,151), followed by Black (29%, n = 923), Asian (26%, n = 835), and Hispanic (9%, n = 275). When pooled across academic years, mean milestone scores did not differ significantly by gender or ethnicity for any of the six competencies. Stratified analyses by academic year and PGY identified some statistically significant differences in individual years; however, these findings were inconsistent in direction and based on small subgroup sizes, limiting interpretability. As such, adjusted mixed-effects models were prioritized for inference.

Across all six competencies, PGY was the strongest and most consistent predictor of milestone scores. Each increase in PGY was associated with an approximately 0.8–0.9-point increase in milestone level (all P<0.001), consistent with expected developmental progression. The number of evaluations completed within the same academic year was positively associated with milestone scores for several competencies. Significant associations were observed for ICS, MK, and PROF, with marginal associations for PC and SBP, while no significant association was observed for PBLI. After adjustment for PGY, evaluation exposure, and academic year, gender and ethnicity were not independently associated with milestone scores for any competency. Academic year fixed effects were generally small and not statistically significant, indicating minimal systematic shifts in milestone scoring across calendar years after accounting for PGY and evaluation exposure. All mixed-effects models demonstrated significant resident-level random intercept variance, indicating meaningful between-resident differences in baseline milestone performance. Likelihood ratio tests confirmed that mixed-effects models provided a significantly better fit than models without random effects (all P<0.001).

In analysis of the qualitative comments submitted in each evaluation, we found no significant difference in the use of masculine language (0.45 vs 0.41, P = 0.55), feminine language (0.85 vs 0.60, P = 0.05), agency (4.63 vs 4.60, P = 0.89), or communion (4.70 vs 4.69, P = 0.97) between evaluations of men and women. We found no difference in word count in evaluations between men and women residents (45.5 vs 46.3, P = 0.75). We found no difference in use of hedging language between men and women (P = 0.59), nor any difference in use of words describing confidence between men and women (P = 0.34).

In the univariable analyses examining the main effects of ethnicity, several differences in language use were observed (Table 3). We found significant word count differences based on ethnicity, where Hispanic residents had fewer words per evaluation than Caucasian residents (36.5 vs 47.3, P = 0.008). We found no significant differences in use of masculine language or agency language by ethnicity. We found significantly less use of feminine language in evaluations of Asian residents (β=–0.31, P = 0.03) compared to Caucasian residents. We also found differences in communion language, with more language of communion in evaluation of Asian residents (β = 1.10, P = 0.001) and Hispanic residents (β = 1.72, P<0.001).

These main effects were then evaluated in adjusted mixed effects models (Table 4). We found no significant differences by gender or ethnicity in use of masculine terms, agency language, hedging language, or confidence language. However, the adjusted analyses continued to demonstrate significantly less feminine language in evaluations of Asian residents (β=–0.27, P = 0.03) after adjustment for resident gender and clustering by resident and faculty member. Similarly, more communal language was observed in evaluations of Asian residents (β = 1.14, P = 0.008) and Hispanic residents (β = 1.74, P = 0.007).

We next examined whether the association between ethnicity and qualitative language differed by resident gender (ie, gender-by-ethnicity interaction effects). We observed no significant interactions for masculine, feminine, agency, hedging, or confidence language. However, we identified a significant interaction for communal language such that Asian women received fewer communal terms than Asian men (β=–1.65, P = 0.03).

This study was the first to analyze qualitative and quantitative evaluations of family medicine residents, looking for differences in faculty-provided milestone ratings as well as in language used in narrative comments with submitted faculty evaluations. Our findings suggest that, while quantitative evaluations do not demonstrate differences in milestone ratings, qualitative evaluations of residents may have some word use differences that are associated with gender and/or ethnicity.

Interestingly, we found no significant differences in quantitative ratings of residents and few qualitative differences across gender or ethnicity. These results are discrepant from prior work in other specialties that have highlighted differences.^3-8 Faculty development bringing awareness to and mitigation of evaluation biases may have reduced discrepancies in quantitative and qualitative evaluation data based on ethnicity or gender. Possibly family medicine faculty training is robust at providing structured and objective evaluative feedback for residents, thus mitigating biases seen in prior studies. Attempts to mitigate bias through faculty development on evaluation best practices (both through professional development by national or regional organizations and the institution itself) also contribute to avoiding biases in qualitative or quantitative evaluation metrics. Additionally, individual institutional CCC structures that could identify discrepant ratings may contribute to these findings. We would be surprised to find, however, that any individual institutional structure captures and mitigates subtle differences in evaluations that would appear over larger sample sizes.

As one exception, we found two differences within data from Asian residents. More communal language was noted in evaluations of Asian women when compared to Asian men. This finding suggests that observed differences in communal language vary based on gender (ie, Asian women have less communal language in evaluations than Asian men in this domain, which is not observed in other races). Additionally, we found that feminine terms were less used in evaluations of Asian residents. Interestingly, this finding did not result in quantitative milestone ratings of Asian residents (as previously reported in internal medicine residents).²² These differences are notable because Asian women in academic medicine have identified stereotyping, feeling othered, and institutional culture as barriers to entry into academic medicine leadership.²³

Possibly our findings among Asian residents are a demonstration of these previously identified themes and representative of barriers to leadership opportunity and growth for Asian women residents. These differences are interesting to consider in the context of the model minority myth (MMM), which downplays structural inequities that Asians face in the United States given a perceived high degree of educational and societal success.²⁴ While the MMM does not fully apply to the gender differences in communal language given that these differences occur within Asian residents alone, the decrease in feminine terminology used in qualitative evaluations of Asian residents is compatible with the stereotypes perpetuated by the MMM. The presence of qualitative differences without quantitative milestone rating discrepancies may be reflective of faculty knowledge of standard progression through milestone ratings throughout residency, while discrepancies in terminology used in qualitative evaluations may indicate a true difference in the evaluation of Asian residents. Also notable is that faculty who do not participate in the CCC process complete resident evaluations, allowing for the possibility that CCC discussion helps to blunt any effects seen by biased comments or evaluations. Finally, some literature has suggested that faculty evaluators may be more reliant on individual writing styles, and style differences may be more impactful on differences in evaluations rather than an experience with an individual resident.²⁵

Our study had several limitations. First, our study was limited to a single site with specific institutional training regarding learner evaluation. As a result, the data may not reflect family medicine faculty evaluations on a larger scale. Additionally, our study period notably came just prior to the emphasis among family medicine professional organizations on implementation of competency-based medical education (CBME) through entrustability and robust frequent observations.²⁶ Possibly biases could be magnified or eliminated with a different approach to evaluation than less frequent and higher-stakes observations and evaluations.

Another limitation included a lack of nonbinary identifying residents in the study population. Our residency management software uses self-identified gender based on residency application data and may not capture all nuance in understanding resident gender identity. Further research including nonbinary identifying individuals and more nuanced data capture regarding gender identity may be useful to detect differences among nonbinary residents compared to women or men counterparts.

Although our analyses accounted for individual evaluator effects using mixed-effects models clustered by rater, we did not examine evaluator demographic characteristics or evaluator-resident gender or ethnicity concordance. We cannot exclude the possibility that some observed differences in language use were influenced by evaluator demographics rather than resident characteristics alone. Finally, we used a narrow definition of comments regarding resident confidence by using words similar to confidence itself. A broader definition could exist that would more fully capture comments referring to resident confidence, but that definition would need further research and validation.

Further research among multiple family medicine residency programs would help determine whether the effects we saw are broadly applicable across family medicine graduate medical education at-large and, if so, provide an avenue to develop training for faculty to mitigate these differences. Additionally, research comparing pre-CBME implementation data with post-CBME implementation data could help to identify unintended deleterious effects of CBME evaluation principles on residents based on gender or ethnic characteristics. Finally, given the explosion of artificial intelligence (AI), one might plausibly expect residency faculty to interact with AI to summarize and generate narrative feedback. Further study regarding the impacts of AI in generating feedback for resident learners and the potential biases that could be introduced will be critical to ensure that fair and equitable evaluations inform resident progression and growth.