Interrater Reliability of a Suture Assessment Tool in Family Medicine Training

Despite the critical role that procedural competence plays in patient outcomes, assessment of technical skills such as suturing remains highly variable.^1,2 While assessments for medical knowledge have become largely objective and reproducible, the evaluation of technical skills continues to depend primarily on subjective faculty impressions.^3,4 In an era where patient safety and clinical outcomes are increasingly tied to the precision of hands-on care, the lack of standardized, validated tools to assess suturing ability represents a serious gap in medical education.^1,2,4

Family physicians frequently perform minor procedures, including suturing, as part of routine practice.⁵ Yet, procedural skill assessment in family medicine remains highly variable, often relying on informal faculty observation rather than structured, objective tools.^6,7 While educators have developed tools aimed at standardizing surgical skill evaluation across surgical specialties, these tools have not been evaluated in family medicine contexts.^2-12 For example, the objective structured assessment of technical skills (OSATS) is a performance-based examination aimed at assessing the clinical competence of surgical residents.^2,8,10 However, its use remains limited by concerns over predictive validity and sparse application outside traditional surgical settings.^3,4 To address these gaps, Sundhagen et al developed an assessment tool specifically for evaluating suturing skills in medical students. Their study demonstrated promising reliability and validity in medical student populations.²

Developing reliable approaches for evaluating suturing performance could enhance both feedback quality and resident skill development. Therefore, the purpose of this study was to evaluate the interrater reliability of the Sundhagen suture assessment tool when applied to family medicine residents. By doing so, we aimed to determine whether the tool may serve as a feasible, objective framework for procedural skills assessment in family medicine residency training.

This study aimed to evaluate interrater reliability (IRR) of the suture assessment tool developed by Sundhagen et al when used by family medicine residency faculty to assess family medicine residents. The study was deemed exempt by the Mayo Clinic Institutional Review Board.

Core faculty from four family medicine residency programs at a single multispecialty institution in the Midwest region of the United States evaluated videos of family medicine residents performing simple interrupted sutures. Participants were eligible for inclusion if they were active core faculty at one of the participating family medicine residency programs and provided consent.

Faculty evaluators independently reviewed 20 de-identified videos of family medicine residents performing simple interrupted sutures. Videos were obtained from routine procedure training sessions at a single residency program, with family medicine residents filmed individually under standardized lighting, consistent camera positioning, and high-resolution recording conditions (Figure 1). To preserve anonymity, only gloved hands were visible, and family medicine residents provided written informed consent for video capture.

Evaluators used a modified version of the Sundhagen suture assessment tool to rate each performance. The original tool includes both subjective assessments and an objective time-based component; however, for this study, only the subjective criteria were assessed, and the time factor was excluded. Evaluators were anonymized using assigned identification numbers. They each received a standard instructional script that outlined the study purpose, explained that the modified Sundhagen tool should be applied exactly as written, and provided logistical guidance for completing ratings in REDCap. This minimal-instruction approach was intentional to evaluate the response process in real-world conditions.¹³

IRR was assessed using Light’s κ for each yes/no checklist item, calculated with the kappam.light function in R (R Foundation). This chance-corrected measure computes the mean Cohen’s κ across all possible pairs of raters. A sample of 15 raters evaluating 20 videos provided greater than 80% power (two-sided α=.05) to detect a difference of 0.2 in reliability estimates. κ values are interpreted as follows: 0 reflects no agreement; 0.01 to 0.20, slight agreement; 0.21 to 0.40, fair agreement; 0.41 to 0.60, moderate agreement; 0.61 to 0.80, substantial agreement; and 0.81 to 1.00, near-perfect agreement.¹⁴

A total of 15 faculty reviewers consented to participate, each rating 20 videos. The Light’s κ estimates ranged from 0.11 to 0.50, indicating fair/moderate IRR for most questions (Table 1). None of the items reached the threshold for substantial agreement (κ≥0.61).

This study aimed to evaluate the IRR of a modified version of the Sundhagen suture assessment tool for use in assessing simple interrupted suturing by family medicine residents. The tool demonstrated fair to moderate IRR, indicating substantial variability in scoring across evaluators. While this variability may limit the tool’s effectiveness as a standardized summative assessment, the tool may still offer value as a structured framework for formative feedback and resident development—pending further refinement before broader implementation.

This study offered several methodological strengths, including the use of prerecorded standardized videos, prior rater training, and an adequately powered design to estimate IRR. However, the study was limited to a single institution, which may affect generalizability. Refining the training protocol with more detailed calibration procedures may have improved the IRR, though such enhancements could also limit generalizability by deviating from typical real-world implementation.

Several factors may have contributed to the limited IRR observed, including ambiguous language in the original assessment tool, which was not modified for this study. The tool may benefit from clearer scoring anchors to guide evaluators in distinguishing between adjacent rating categories. Inconsistent scoring may also have resulted from vague criteria for items such as “correct forceps use” or “damage to the suture model.” Enhancing the tool by providing visual exemplars and explicit definitions may also improve consistency. Future studies also may explore improving IRR by providing structured calibration sessions using annotated videos and consensus discussions.¹⁵

This study found that a structured assessment tool for evaluating simple interrupted suturing by family medicine residents demonstrated fair to moderate IRR, reflecting variability in scoring across evaluators. These findings suggest that while the tool may be useful for guiding formative feedback for family medicine residents, the tool shows limited utility for summative or standardized assessments. Further refinement—including behaviorally anchored scoring criteria, standardized rater training, and integration into structured formats—may improve reliability and support the tool’s broader application in family medicine residency education.