Screening for Conscientiousness at Start of Residency: Early Detection of Professionalism Concerns

Christopher M. Haymaker, PhD | Kristi VanDerKolk, MD | Megan Rhodes, BA | Jeff Young, MS | Lisa Graves, MD, MClSc

PRiMER. 2024;8:28.

Published: 4/23/2024 | DOI: 10.22454/PRiMER.2024.733332


Introduction: Early identification of residents who may require remediation could help prevent problems for both individual residents and their programs. Our aim in this project was to identify whether residents prone to challenges with professionalism could be identified early.  

Methods: For 3 years we tracked onboarding tasks completed by residents between official match and the first day of residency to develop a tool that would help identify residents who may be at risk for problems with timely chart completion and subsequent remediation. We compared residents’ “at risk” scores with in-training exam (ITE) scores to differentiate between professionalism (task completion issues) and medical knowledge.

Results: Our at-risk tool successfully predicted timely chart completion rates and need for remediation within our residency program (Fischer’s exact P<.001), but was not predictive of ITE scores.

Discussion: Using readily accessible information, we can identify residents at risk for poor task completion performance, who may benefit from extra support and development in the area of organizational skill. Early identification may increase opportunities for early intervention.


In residency, learners who struggle to meet deadlines consume significant time and resources. Medical students who are cited by their school for inappropriate behavior are more likely to have similar problems with professionalism in residency and tend to engage in similar inappropriate behavior postgraduation.1 These learners require reminders, respond slowly to feedback, and are more likely to continue similar patterns.2,3 Insensitivity to deadlines on the part of residents can adversely affect patient care and whether programs meet graduate medical education (GME) requirements.

Residents who struggle with documentation and other professionalism-related tasks are at increased risk professionally. The increased demands of residency may stretch learners with less developed organizational skills.4 Learners who struggle with deadlines often face remediation. Scholarship in emergency medicine has identified a variety of best practices for residency, including intervention strategies for poor attendance and missed deadlines.5,6 Such modest offenses as late documentation predict unprofessional behavior years later including reports to the licensing board and loss of license.7,8

Our aim in this project was to identify whether residents prone to challenges with professionalism can be identified early. We hypothesized that late completion of onboarding documents preresidency would predict problems with timely chart completion and the subsequent need for remediation. We also explored medical knowledge (by in-training examination) as an outcome to assess the distinctiveness of the relationship of task completion to professionalism.


Our family medicine residency has nine residents per year. We reviewed deidentified archival data for three cohorts of residents (n=27).

Our program coordinator tracks preonboarding tasks necessary to begin residency and sends reminders to incoming residents to ensure timely completion (Table 1). Our at-risk index includes a 3-point rating for each task: (2) early, (1) on-time, and (0) late. Our coordinator also rates each incoming resident on ease of communication: (2) responds immediately, (1) timely response, or (0) requires multiple prompts. Total at-risk index scores range from 0 to 10 (four task-related scores ranging from 0-8 and the coordinator rating ranging from 0-2). Items were highly correlated, with simple correlations between items ranging from 0.6 to 0.95 (mean correlation=0.79)

Chart completion within 2 days of an encounter is our institutional standard of patient care and professional communication in primary care clinics, including prebuilt reports for measuring clinician performance. For the purposes of this study, we used average chart completion during the first year as a marker of successful skill development. The distribution of resident scores appeared bimodal, with a natural break at 95%, which we chose as a cut score.

We recorded each resident’s remediation status as present or absent. Present remediation status included any resident who required intervention to meet performance requirements for patient care or professionalism as determined by the Clinical Competency Committee.

The In-Training exam (ITE) is a family medicine-specific standardized test developed by the American Board of Family Medicine taken annually by all residents in our program. Scores are predictive of passing the Family Medicine Certification Examination.9


The Institutional Review Board at Western Michigan Homer Stryker MD School of Medicine determined that this study met the criteria for exempt status (November 2022). Archival data were deidentified and analyzed using SAS Proprietary Software version 9.4 (SAS Institute, Cary NC). We examined the predictive estimates of our total at-risk index using Fischer’s exact test and a cut score of 3 to identify residents at risk of remediation. Examination of the distribution of at-risk scores suggested a bimodal distribution with a natural break between scores of 3 or less and scores of 5 or more. This allowed us to calculate positive and negative predictive values using remediation status categorically.


The at-risk index scores ranged from 0 to 10, with a mean of 4.67, median of 5, and standard deviation of 2.8. The at-risk index was predictive of timely chart completion (Table 2) and remediation (Table 3).

The positive predictive value (PPV), negative predictive value (NPV), sensitivity and specificity of both ratings to identify learners at risk for problems with chart completion and need for remediation indicate good utility with few false positives and negatives in either category (Tables 2 and 3). Using a single item of the at-risk index (eg, coordinator rating only) yielded identical results.

ITE scores for postgraduate year 1 (PGY1) residents ranged from 290 to 490, with a mean of 382 and standard deviation of 60. Neither at-risk index scores nor coordinator rating were predictive of PGY1 ITE scores (P<.40)


Our onboarding assessments (ie, at-risk index and coordinator rating) were strong predictors of later chart completion and remediation outcomes. While the tasks monitored are program specific, many institutions require similar administrative task completion prior to beginning residency. Monitoring completion of these tasks early in residency may be a useful indicator of later professionalism issues, and brings additional objectivity to what could be a more subjective assessment by a program coordinator.

With a robust at-risk index, programs may be able to offer explicit training for time-sensitive tasks and establish clear expectations for frequent follow-up with residents who may be at-risk in this domain. Early intervention has the potential to prevent the need for remediation.

With a small, single-institution sample, we are cautious about generalizing these results or using them as a sole indicator of professionalism. Use of this screening to identify early concerns should be used compassionately and with careful attention to potential bias (eg, people may miss deadlines for a host of reasons, yet educators can inadvertently ascribe inaccurate motives or personality attributes when this occurs in the absence of better data about context). We recommend that these data be used to elicit more information from residents about barriers to task completion once they begin residency, and to identify potential supports for residents who have struggled in this domain.

Careful thought should be given to the cut score to identify residents for early intervention, balancing the risk of false positives and negatives and taking residency context into account. The accessibility of these data makes screening easy to implement and enables targeted early intervention.


  1. Krupat E, Dienstag JL, Padrino SL, et al. Do professionalism lapses in medical school predict problems in residency and clinical practice? Acad Med. 2020;95(6):888-895. doi:10.1097/ACM.0000000000003145
  2. Kelly M, O’Flynn S, McLachlan J, Sawdon MA. The clinical conscientiousness index: a valid tool for exploring professionalism in the clinical undergraduate setting. Acad Med. 2012;87(9):1218-1224. doi:10.1097/ACM.0b013e3182628499
  3. McLachlan JC, Finn G, Macnaughton J. The conscientiousness index: a novel tool to explore students’ professionalism. Acad Med. 2009;84(5):559-565. doi:10.1097/ACM.0b013e31819fb7ff
  4. Armstrong AD, Agel J, Beal MD, et al. Use of the behavior assessment tool in 18 pilot residency programs. JBJS Open Access. 2020;5(4):e20.00103. doi:10.2106/JBJS.OA.20.00103
  5. Regan L, Hexom B, Nazario S, Chinai SA, Visconti A, Sullivan C. Remediation methods for milestones related to interpersonal and communication skills and professionalism. J Grad Med Educ. 2016;8(1):18-23. doi:10.4300/JGME-D-15-00060.1
  6. Williamson K, Quattromani E, Aldeen A. The problem resident behavior guide: strategies for remediation. Intern Emerg Med. 2016;11(3):437-449. doi:10.1007/s11739-015-1367-5
  7. Papadakis MA, Hodgson CS, Teherani A, Kohatsu ND. Unprofessional behavior in medical school is associated with subsequent disciplinary action by a state medical board. Acad Med. 2004;79(3):244-249. doi:10.1097/00001888-200403000-00011
  8. Stern DT, Frohna AZ, Gruppen LD. The prediction of professional behaviour. Med Educ. 2005;39(1):75-82. doi:10.1111/j.1365-2929.2004.02035.x
  9. O’Neill TR, Li Z, Peabody MR, Lybarger M, Royal K, Puffer JC. The predictive validity of the ABFM’s in-training examination. Fam Med. 2015;47(5):349-356. Accessed February 9, 2024.

Lead Author

Christopher M. Haymaker, PhD

Affiliations: Western Michigan Homer Stryker MD School of Medicine, Kalamazoo, MI


Kristi VanDerKolk, MD - Western Michigan University Homer Stryker M.D. School of Medicine, Kalamazoo, MI

Megan Rhodes, BA - Western Michigan Homer Stryker MD School of Medicine, Kalamazoo, MI

Jeff Young, MS - Western Michigan Homer Stryker MD School of Medicine, Kalamazoo, MI

Lisa Graves, MD, MClSc - Western Michigan University Homer Stryker M.D. School of Medicine, Kalamazoo, MI

Corresponding Author

Christopher M. Haymaker, PhD

Correspondence: Western Michigan Homer Stryker School of Medicine, 1000 Oakland Drive, Kalamazoo, MI 49008. 269-349-2641, ext. 439. Fax: 269-488-8978.


Fetching other articles...

Loading the comment form...

Submitting your comment...

There are no comments for this article.

Downloads & Info


Related Content