Resident Competency in Pelvic Exam Skills Not Predicted by Early Assessment

Accreditation Council for Graduate Medical Education (ACGME) Milestones Project requirements include competency-based evaluation of resident physicians in many areas, including pelvic examination.¹ Although the US Preventative Services Task Force found insufficient evidence to assess the balance of risks and harms of screening pelvic examinations in asymptomatic, nonpregnant women, pelvic examination is essential for evaluation of abdominal, urologic, and pelvic symptoms.²

Training and assessment is expensive and time consuming. Residents matriculate with varied levels of prior experience. Prior to initiating formal pelvic examination training, our residency received reports from staff, residents, and faculty that some residents were not comfortable performing pelvic examinations. In response, we created a training program for residents to improve their skills using gynecology teaching assistants (GTAs). Briefly, a GTA is a trained layperson who allows a learner to examine her in order to provide feedback regarding the learner’s bimanual, speculum, and communication skills. Although programs using GTAs to teach pelvic examination skills to medical students have demonstrated benefit, no published studies have evaluated the use of GTAs with resident physicians.^3-7 Our goal was to accurately predict residents who required additional pelvic examination training based on results of early skill evaluation and/or demographic characteristics of residents (such as self-reported exam experience or gender, since anecdotally we found female residents had more pelvic exam experience in medical school). Our hypothesis was that the quality of early performance of one or more microskills would predict subsequent performance and allow residency faculty to better allocate training resources.

Incoming residents self-identified learning needs related to pelvic examinations. Residents performed two pelvic examinations with GTAs who were aware of their learning needs. During intern year, the residents were subsequently observed by residency faculty while performing pelvic examinations for at least three continuity clinic patients. Faculty and GTAs evaluated residents using a checklist of microskills that included domains of communication, planning, and dexterity.⁸ Additionally, faculty provided a global assessment of the residents’ competency in performing each pelvic examination. Each resident required three satisfactory evaluations by faculty before they were considered by the residency program as competent in performing pelvic exams without supervision. Serial faculty evaluations were halted when a resident’s overall performance during three exams was rated as competent. Exploratory linear regression analysis of resident demographics and assessment results was performed using an Excel-based software tool (described in Braun and Oswald in 2011, and available by direct download from this author at http://dl.dropbox.com/u/2480715/ERA.xlsm?dl=1).⁹ This model performs ordinary least squares (OLS) regression on the (2^P-1) possible combination of predictors and produces a number of indicators of predictor importance.

Results of this analysis were used to create a model in which early resident performance predicted subsequent competency. To quantify the success of each resident in performing the pelvic exam in clinic, a Ready Ratio (RR) was calculated. The Ready Ratio is defined as the number of successful faculty evaluations divided by the total number of evaluations for each resident. The potential values for RR are limited to combinations of the number of successful evaluations divided by the total number of evaluations.

This research was granted an exemption from formal review by the University of Wisconsin School of Medicine and Public Health institutional review board.

Sixty eight residents were evaluated by 67 faculty members from 2009 to 2014 in a US family medicine residency program with five continuity clinic sites in urban, suburban, and rural locations. The number of evaluations needed to qualify as competent ranged from three to eight. More than half of the residents (56%) required more than the minimum of three observed pelvic exams (Table 1). The amount of time between the residents’ GTA examinations and the completion of their minimum of three observed pelvic exams with clinic patients ranged from 1 to 12 months. Most of the faculty who performed evaluations (69%) performed between one and four evaluations over a 5-year period (Table 1).

Ready Ratios ranged from 0.43-1.0. These values were in a negatively skewed distribution; commonly used transformations (eg, square root) could not remedy this.

The Braun and Oswald software tool⁹ generated 511 possible regression analysis models involving nine (limit of the tool) microskill predictors with the highest regression weights and indices of model adequacy (incremental R² [change in coefficient of determination], general dominance weights, and relative importance weights).

We included demographic information such as gender and clinic site in possible models because (1) anecdotally we found incoming female residents completed more pelvic exams during medical school, and (2) our residency clinic sites were varied in location and patient population. We included the date that the first pelvic examination was evaluated in clinic because we postulated that an early exam would suggest interest in this skill and/or opportunity to apply what was learned with the GTAs. However, neither the items above nor the residents’ average score given by the GTAs accurately predicted RR.

The checklist items most likely to be missed (and therefore the items that most accurately predicted the RR) were “identifies the cervix” (unstandardized coefficient=0.730 [P<.001, 95% confidence interval {CI}=0.351 to 1.110]), and “properly assembles equipment” (unstandardized coefficient=0.618 [P=.004, 95% CI = 0.210 to 1.026]). The model with these two checklist items had the best indices of adequacy as defined by the software tool. This model was 100% sensitive (95% CI=0.859-1.000) but only 36.8% specific (95% CI= 0.223-0.540) in predicting a high RR. In addition, these two items explained only eight percent of the variance in RR. See Table 2 for calculation of sensitivity, specificity, and accuracy of our model.

Our model based on early skills assessments was not sufficiently specific to identify residents who would need more extensive training to achieve competency in performing pelvic examination.

The results show that 44% of residents completed all supervised examinations successfully. This observed high pass rate does not coincide with anecdotal stories of former residents struggling with the pelvic exam. There are several possible explanations for this discrepancy. The current residents could be highly skilled, or we may have underestimated the abilities of former residents. Our tool did not assess residents’ self-confidence in performing this exam. More likely, however, the current faculty or the checklist method of evaluation were not sufficiently discriminating. Since many faculty evaluate only a few examinations per year, they may require more training to be able to distinguish competent residents from those who require more training. Additionally, even for residents who were not judged to be competent on any one evaluation, most individual microskill checklist items were noted to be completed successfully. This suggests evaluators were using criteria other than the microskills (and not captured in the current checklist) to determine competency.

Our study was limited by the lack of a control group (ie residents who did not receive training by GTAs). The specificity of our model would likely be improved by (1) standardizing faculty evaluations through faculty development, (2) qualitative interviews of faculty performing many evaluations to determine criteria not currently on the checklist that they use to determine competency, and/or (3) using a smaller number of faculty to perform the assessments. Additionally, assessment of an intern’s learning needs versus assessment of competency (later in residency) may require different processes or tools.

A predictive model based on early skills assessments by gynecological teaching assistants and family medicine faculty was not sufficiently specific to identify residents who would need more extensive training to achieve competency in performing pelvic examination. Next steps could include limiting the number of faculty assessors, faculty development to improve discriminatory capacity, and/or creating separate processes for (1) providing feedback and identifying learning needs in new residents, and (2) documenting competency in performance of pelvic exams.