Imprecise Clinical Assessments and Inaccurate Grades: Family Medicine Clerkship Director Perspectives

Medical school core clerkship grades are major influencers for residency program directors (PDs) in the selection of student applicants.^1-3 With the United States Medical Licensing Examination Step 1 now a Pass/Fail exam, clerkship grades have further risen in significance for PDs.⁴ Clinical evaluations by faculty, often in the form of subjective assessments, are heavily weighted in clerkship grading rubrics.⁵ Studies are mixed whether these clinical ratings are reflective of actual knowledge when compared to written and oral exams. ^6- 8 Clinical evaluations can be influenced by preceptors’ habits, comparisons to other students, inflation pressures, and students’ personalities. ⁹

Significant variation in grading within and among medical schools has led to growing student concern nationwide about unfair grading from imprecise and inaccurate clinical ratings.¹⁰ One measure of student concern, the grade challenge rate, has been described in other clerkships as ranging from 4.5% to 8.0% of students in a given year.^{11, 12} Family medicine clerkships may be especially subject to student concerns about fairness, given the diversity of academic, community, and volunteer faculty who perform clinical assessments. Grade inflation has been identified as a particular challenge for family medicine clerkship directors (FMCDs). ¹³

No recent studies have reported on the prevalence of imprecise clerkship raters or FMCD’s satisfaction with their current grading systems. Our study aimed to reveal associations among FMCD’s perceived prevalence of improper faculty ratings, confidence in grading systems, length of tenure, and comfort level with addressing student dissatisfaction. We hypothesized that many FMCDs share concern regarding imprecise faculty ratings.

Data were gathered and analyzed as part of the 2023 Council of Academic Family Medicine (CAFM) Educational Research Alliance (CERA) survey of FMCDs. CAFM is a joint initiative of four major academic family medicine organizations: the Society of Teachers of Family Medicine, the North American Primary Care Research Group, the Association of Departments of Family Medicine, and the Association of Family Medicine Residency Directors. The general methodology of the annual questionnaire, termed an omnibus survey, has been previously described. ¹⁴

CERA distributed the survey via email to 169 (154 US and 15 Canadian) FMCDs between May 2023 and June 2023. The specific methodology for the 2023 survey has been described in detail.¹⁵ The invitation email included an explanation and link to the online survey, which was conducted via SurveyMonkey (SurveyMonkey Inc). Nonrespondents received three requests to complete the survey.

The survey featured a set of demographic questions to determine characteristics of the FMCDs and their clerkships. We submitted 10 additional closed-response survey items (Table 1) that assessed FMCD perceptions of imprecise grading, current and preferred grading systems, and comfort with correcting aberrant preceptor grading.

The first survey item established the perceived prevalence of imprecise preceptor ratings. FMCDs rated their confidence that preceptors would give two students of identical competence the same clinical evaluation rating (1=not at all confident, 5=completely confident). Respondents also rated the percentage of evaluators suspected of inappropriately rating student performance (0%-100%). FMCDs approximated the percentage of students that had expressed concern over their preceptor ratings within the last calendar year (0%-100%).

Next, FMCDs reported their current and preferred grading systems based on the options recognized by the Association of American Medical Colleges.¹⁶ Subsequently, we asked for the percentage weight of clinical evaluations within grading rubrics (0%-100%) and whether FMCDs had recently adjusted that weight (multiple-choice item). Two items inquired whether grading systems accounted for outlier ratings (multiple-choice item) and which factors likely influenced preceptors’ ratings (multiple-choice item). A final item assessed how comfortable FMCDs felt with remediating preceptors who inappropriately rate students (1=very uncomfortable, 5=very comfortable).

We analyzed anonymous data with SPSS Version 26 (IBM) using descriptive statistics and Pearson correlations. We used Kruskal-Wallis H tests and Mann-Whitney U tests to determine the statistical significance of differences among respondent selections.

The American Academy of Family Physicians Institutional Review Board approved the study.

A total of 88 of 169 FMCDs (52%) responded to at least two survey items, and 85 FMCDs completed all 10 items. Of the respondents, 60% (n=53) were women and 68% were White (n=60). The average number of years spent in their current FMCD role was 6.64±5.08 (Table 2).

We found that 53% (n=47) of FMCDs reported they were “not,” “slightly,” or “somewhat confident” their preceptors would give two students of identical competence the same clinical rating. Another 40% (n=35) were “fairly confident,” and only 7% (n=6) of FMCDs were “completely confident” on this item.

FMCDs estimated that an average of 38% of their preceptors inaccurately rated student performance. Almost all respondents (n=86, 99%) perceived that at least 5% of their preceptors were overrating or underrating students. FMCDs estimated that 8% (SD=10%; range: 0-50%) of students expressed concern over their clinical ratings within the last year. We found no relationship between the number of years as FMCD and prevalence of perceived inaccurate assessors (Table 3). Similarly, years of experience as a FMCD was not correlated with estimated percentage of dissatisfied students.

Inadequate faculty development (n=56, 64%), pressure to bestow high grades (n=55, 63%), and a fear of consequences to the preceptor (n=52, 59%) were selected as likely influencers of imprecise ratings. Of respondents, 14% (n=12) felt that none of these factors influenced their preceptors. Respondents who selected any of the potential above-mentioned influencers on Item 9 reported a higher perceived prevalence of imprecise grading on Item 2, compared to FMCDs who selected no influencers (P=.023, r=0.24; Table 3). Notably, the same CDs who believed that any of the three factors impacted their faculty estimated a higher percentage of students expressing concern over their clinical ratings (P<.001, r=0.38) compared to respondents denying the influences (Table 3).

The average weight of clinical evaluations in clerkship grading rubrics was 57% (SD=21%; range: 0&-100%). Most FMCDs (n=72, 84%) had not decreased the percentage weight of clinical evaluations within their rubrics over the last 3 years. The majority (n=62, 70%) of FMCDs reported that their grading systems do not automatically account for outlier preceptor ratings.

Of respondents, 57% (n=50) used an Honors/High Pass/Pass/Fail grading system (Table 4). We identified that 51% (n=44) of FMCDs preferred a different grading system than what they currently used. The two grading systems that were most often selected were Pass/Fail (n=29, 33%) and Honors/Pass/Fail (n=27, 31%).

FMCDs with four grading tiers estimated a higher percentage of inaccurate clinical evaluations compared to FMCDs with three tiers. The estimated percentage of inaccurate clinical rating varied significantly based on the school’s grading system (P<.001; Table 3). The estimated percentage of students dissatisfied with their preceptor ratings also varied significantly based on the school’s grading system (P=.003). Namely, FMCDs using four grading tiers estimated more student complaints about grading compared to FMCDs using two or three tiers.

Of FMCD respondents, 55% (n=48) were either “somewhat” or “very comfortable” addressing and correcting preceptors who inappropriately rated students’ clinical performance. We found no difference in comfort with remediating preceptors based on years of experience as an FMCD (Table 3).

One-half of responding FMCDs were not confident that their preceptors would give two students of identical competence the same clinical evaluation rating. Inadequate faculty development was most often selected by respondents as a contributor to improper ratings. We postulate that many FMCDs feel that adequate and effective faculty development is limited by the intuitiveness and specificity of their institutional assessment tool.

The estimated percentage of students challenging their clinical ratings was consistent with other clerkships reported in the literature.¹² Despite FMCD awareness of inaccurate ratings, the percentage weight of clinical evaluations in clerkship grading has remained high. More grading tiers correlated with higher estimates of imprecise graders and higher estimates of student dissatisfaction. Nonetheless, FMCDs remained mixed in preferring tiered grading systems versus a Pass/Fail system. Possible reasons for preferring tiered grading may relate to residency stakeholders and considerations external to the clerkship.

This study gathered perceptions of issues facing clerkships, students, and preceptors. While respondents could provide firsthand views on many survey items, several items required FMCDs to approximate information secondhand.

This 10-question survey explored only upstream causes and downstream effects of aberrant preceptor clerkship grading. We did not differentiate the prevalence of over- versus underrating of student performance by preceptors. Open-ended questions and matrix formats were not permitted in this survey. Narrative input on strategies to address imprecise grading was not gathered. While grade inflation and fear of retribution may be important factors in clinical assessments, these factors were not specifically measured for impact or influence.

Interestingly, FMCDs with more grade options reported a higher frequency of student complaints. While Pass/Fail grading systems may reduce the harms of competitive stress and evaluator bias, a concurrent need exists for multidimensional assessments of students’ strengths and weaknesses. The growing popularity of competency-based medical education encourages the use of directly observed assessments with behaviorally anchored rubrics. Criterion-driven tools, such as workplace-based assessments may reduce student perceptions of subjectivity; however, the de-emphasis on norm-based assessments may increase PDs’ difficulty in ranking and stratifying students in effective and equitable ways.

Essentially all responding FMCDs perceived some level of improper faculty ratings in their courses. Despite clinical evaluation scores being the most heavily weighted component of their clerkship, a minority of FMCDs reported a systematic mechanism within their grade calculation process to account for outlier clinical ratings. In an effort to lessen the impact of inaccurate grading, medical schools in the United States and Canada should continue to explore ways to reduce subjectivity within their clinical assessment tools. Given the inherent challenges to delivering faculty development to busy clinical preceptors, clinical rating anchors based on directly observed behaviors may improve rating precision.