Screening for Conscientiousness at Start of Residency: Early Detection of Professionalism Concerns

In residency, learners who struggle to meet deadlines consume significant time and resources. Medical students who are cited by their school for inappropriate behavior are more likely to have similar problems with professionalism in residency and tend to engage in similar inappropriate behavior postgraduation.¹ These learners require reminders, respond slowly to feedback, and are more likely to continue similar patterns.^2,3 Insensitivity to deadlines on the part of residents can adversely affect patient care and whether programs meet graduate medical education (GME) requirements.

Residents who struggle with documentation and other professionalism-related tasks are at increased risk professionally. The increased demands of residency may stretch learners with less developed organizational skills.⁴ Learners who struggle with deadlines often face remediation. Scholarship in emergency medicine has identified a variety of best practices for residency, including intervention strategies for poor attendance and missed deadlines.^5,6 Such modest offenses as late documentation predict unprofessional behavior years later including reports to the licensing board and loss of license.^7,8

Our aim in this project was to identify whether residents prone to challenges with professionalism can be identified early. We hypothesized that late completion of onboarding documents preresidency would predict problems with timely chart completion and the subsequent need for remediation. We also explored medical knowledge (by in-training examination) as an outcome to assess the distinctiveness of the relationship of task completion to professionalism.

Our family medicine residency has nine residents per year. We reviewed deidentified archival data for three cohorts of residents (n=27).

Our program coordinator tracks preonboarding tasks necessary to begin residency and sends reminders to incoming residents to ensure timely completion (Table 1). Our at-risk index includes a 3-point rating for each task: (2) early, (1) on-time, and (0) late. Our coordinator also rates each incoming resident on ease of communication: (2) responds immediately, (1) timely response, or (0) requires multiple prompts. Total at-risk index scores range from 0 to 10 (four task-related scores ranging from 0-8 and the coordinator rating ranging from 0-2). Items were highly correlated, with simple correlations between items ranging from 0.6 to 0.95 (mean correlation=0.79)

Chart completion within 2 days of an encounter is our institutional standard of patient care and professional communication in primary care clinics, including prebuilt reports for measuring clinician performance. For the purposes of this study, we used average chart completion during the first year as a marker of successful skill development. The distribution of resident scores appeared bimodal, with a natural break at 95%, which we chose as a cut score.

We recorded each resident’s remediation status as present or absent. Present remediation status included any resident who required intervention to meet performance requirements for patient care or professionalism as determined by the Clinical Competency Committee.

The In-Training exam (ITE) is a family medicine-specific standardized test developed by the American Board of Family Medicine taken annually by all residents in our program. Scores are predictive of passing the Family Medicine Certification Examination.⁹

The Institutional Review Board at Western Michigan Homer Stryker MD School of Medicine determined that this study met the criteria for exempt status (November 2022). Archival data were deidentified and analyzed using SAS Proprietary Software version 9.4 (SAS Institute, Cary NC). We examined the predictive estimates of our total at-risk index using Fischer’s exact test and a cut score of 3 to identify residents at risk of remediation. Examination of the distribution of at-risk scores suggested a bimodal distribution with a natural break between scores of 3 or less and scores of 5 or more. This allowed us to calculate positive and negative predictive values using remediation status categorically.

The at-risk index scores ranged from 0 to 10, with a mean of 4.67, median of 5, and standard deviation of 2.8. The at-risk index was predictive of timely chart completion (Table 2) and remediation (Table 3).

The positive predictive value (PPV), negative predictive value (NPV), sensitivity and specificity of both ratings to identify learners at risk for problems with chart completion and need for remediation indicate good utility with few false positives and negatives in either category (Tables 2 and 3). Using a single item of the at-risk index (eg, coordinator rating only) yielded identical results.

ITE scores for postgraduate year 1 (PGY1) residents ranged from 290 to 490, with a mean of 382 and standard deviation of 60. Neither at-risk index scores nor coordinator rating were predictive of PGY1 ITE scores (P<.40)

Our onboarding assessments (ie, at-risk index and coordinator rating) were strong predictors of later chart completion and remediation outcomes. While the tasks monitored are program specific, many institutions require similar administrative task completion prior to beginning residency. Monitoring completion of these tasks early in residency may be a useful indicator of later professionalism issues, and brings additional objectivity to what could be a more subjective assessment by a program coordinator.

With a robust at-risk index, programs may be able to offer explicit training for time-sensitive tasks and establish clear expectations for frequent follow-up with residents who may be at-risk in this domain. Early intervention has the potential to prevent the need for remediation.

With a small, single-institution sample, we are cautious about generalizing these results or using them as a sole indicator of professionalism. Use of this screening to identify early concerns should be used compassionately and with careful attention to potential bias (eg, people may miss deadlines for a host of reasons, yet educators can inadvertently ascribe inaccurate motives or personality attributes when this occurs in the absence of better data about context). We recommend that these data be used to elicit more information from residents about barriers to task completion once they begin residency, and to identify potential supports for residents who have struggled in this domain.

Careful thought should be given to the cut score to identify residents for early intervention, balancing the risk of false positives and negatives and taking residency context into account. The accessibility of these data makes screening easy to implement and enables targeted early intervention.