Measuring Clinical Preparedness After Residency Training: Development of a New Instrument

Residency training is designed to prepare physicians for independent clinical practice. However, studies have indicated that not all residents feel prepared to practice independently. ^{1, 2} A 1998 national survey study assessed 2,626 residents at the time of training completion in internal medicine, pediatrics, family practice, and other specialties. ¹ More than 10% of residents in each specialty reported feeling unprepared to undertake one or more tasks common in their disciplines. Much has happened in graduate medical education since these early studies, including the development and implementation of the Accreditation Council of Graduate Medical Education (ACGME) general competencies, ³ ACGME milestones, now in its second iteration, ⁴ and a recent movement toward competency-based graduate medical education. ⁵

Recent studies have revisited preparedness for independent clinical practice in various disciplines, including family medicine. ^6-10 Collectively, these studies continue to report that graduates of residency training appear not fully prepared for independent clinical practice. A few caveats deserve mention. Some of these studies were conducted outside the United States. ^{9, 10} Also, the vast majority of these studies used resident self-reported or program directors’ survey findings close to the time of residency graduation. ^{6, 7, 9, 10} Survey studies use subjective measures, often affected by different types of biases. Program directors may be affected by social response bias, to avoid being perceived as graduating residents not fully prepared. Several survey studies of physicians have revealed that self-reporting or recall bias affects findings, where inflation or deflation of their perceived performance exists. ^{11, 12} What is needed are more objective measures of assessment to assess preparedness more accurately for independent practice.

In family medicine, questions about the length of training have been debated for more than 2 decades, ^{13, 14} leading the American Board of Family Medicine Foundation to fund the Length of Training Pilot (LoTP). ¹⁵ A core question this pilot study was designed to explore was the extent to which length of training affects preparedness for independent clinical practice. The purpose of this paper is to report on the development and pilot testing process for two new instruments and to publish versions that may be beneficial to other graduate medical education researchers.

The LoTP (2013-2023) was a mixed methods prospective case-control pilot study designed to explore several learner outcomes related to the length of training, including scope of practice, preparedness for independent practice, and clinical knowledge. ¹⁵ A number of published papers related to this study can provide additional background. ^16-20 Briefly, 17 residency programs in good standing with ACGME and who agreed to participate in required evaluation activities were selected to participate. All evaluation activities were overseen by researchers in the Department of Family Medicine at Oregon Health & Science University (OHSU). All LoTP programs obtained local institutional review board (IRB) approval, and OHSUs IRB granted an educational exemption to obtain data from study sites (IRB #9770).

Figure 1 illustrates the timeline for instrument development and testing. The LoTP evaluation team began working on instrument development in the Spring of 2015, initially conducting a literature review on existing studies that assessed preparedness for independent clinical practice. To avoid use of resident self-reporting or program directors’ assessments, we sought to develop more objective measures that would be feasible to deploy and sensitive enough to measure differences in clinical preparedness between 3-year and 4-year training models. We defined clinical preparedness as, “The extent to which graduates of family medicine residency training are independent/self-reliant in practicing core skills in the care of patients.” We also determined that the settings the care was to be provided in needed to be comprehensive, including outpatient, inpatient, and other (home, long-term care facilities, specialty care facilities). This definition was used to introduce the survey to expert observer respondents.

We conducted eight key informant interviews with nine rural and urban family physicians between May and June of 2015 regarding how best to assess recent graduates of family medicine residency training. One interviewee was from a solo practice; two were from small family medicine clinics, one of which was from a multigroup health system; three were from federally qualified health centers (FQHCs); and two were private practices, one of which was self-owned and the second was not. Table 1 shows the key interview questions along with key findings. In summary, we learned that significant variability exists in the onboarding of new physicians joining a practice. Typically, new hires, including new graduates, have no supervising physician. Direct observation is inconsistent, and indirect observations are more likely to be the primary source of information about preparedness.

While findings from the interviews were helpful, we decided to convene leaders in family medicine, expert evaluators, and other stakeholders to take part in a Clinical Preparedness Measurement Summit, held in Portland, Oregon in September of 2015, to guide decisions related to measuring preparedness for independent clinical practice. The 1-day meeting included representatives from the LoTP Executive Committee, the American Board of Family Medicine (ABFM), the Society of Teachers of Family Medicine (STFM), the American Academy of Family Physicians (AAFP; including a representative from the AAFP Commission on Education), the Association of Family Medicine Residency Directors (AFMRD), the Residency Review Committee (RRC) members, both allopathic and osteopathic physician representatives, and both rural and urban residency training program directors. Seventeen stakeholders and seven members of the LoTP evaluation team from OHSU attended.

The goals of the summit were (a) to engage measurement experts, leaders in family medicine education, and key stakeholders in defining key areas of clinical preparedness for independent practice; and (b) to decide on key measurement and analytic approaches related to specified measurement issues associated with length of training. The summit provided background on the LoTP, current state of the literature on measuring clinical preparedness, results from the key informant interviews, proposed outcome variables and analytic covariates, information on how to account for threshold setting among expert observers (eg, easy scorers vs hard scorers), and pilot testing plans.

We decided to use the then recently published (2015) entrustable professional activities (EPAs) for family medicine end-of-residency training as the framework for assessment of clinical preparedness by physician assessors. ²¹ For the instrument designed for staff assessors, we identified a set of nine validated variables in published literature that assessed interpersonal communication skills ²² and a set of seven validated variables designed to measure processes of care delivery ²³ by staff members working with residents during patient care.

To address the issue of threshold setting, or the tendency to assign ratings of performance that are different from the ratings that the performance warrants, which has been well-documented in several disciplines, ²⁴ we developed a series of clinical scenarios for both the physician assessor and the staff assessor to determine how they set their threshold (eg, are they hard or easy raters) when determining readiness for independent clinical practice. We specifically developed and then discussed the scenarios during the summit and found during pilot testing that participants produced the desired range in thresholds that would allow us to account for this issue in analyses. The final draft surveys contained 61 items for the physician assessor and 36 items for the staff assessor in the categories described in Table 2.

Finally, we decided on the best timing to administer these two surveys. Recognizing the reality of on-the-job training, we chose to assess the graduates when they were fully oriented to their new positions but before they began to learn on the job. Thus, after an in-depth discussion, we came to consensus to administer the surveys 3 months after the new graduates started their first position as an independent clinician.

After constructing the final draft surveys, we recruited five clinics in a variety of both rural and urban settings because the LoTP sites were located in both urban and rural settings. Further, the pilot sites included a large health system clinic, local health department clinic, residency training continuity clinic, a clinic that does not train residents, and an FQHC clinic. We visited these clinics, typically during lunch, which we provided, and asked volunteer physicians and staff members to complete the surveys about the last physician who joined their practice. We assessed how long these volunteers took to complete the surveys and then used cognitive interviewing techniques ²⁵ to assess whether they were responding to the questions as we intended and to determine whether the order of questions influenced their responses. After each session, the surveys were revised and the process was repeated. By the final pilot test session, no further revisions were needed. Both surveys, along with the respective scales used to assess variables, are included in the appendixes (Appendix A, physician assessor; Appendix B, staff member assessor).

After the first year of data capture (2016), we assessed frequencies of all variables to determine the extent that the survey was capturing a range in physician performance and to be sure that the survey could discriminate readiness for independent clinical practice between 3 and 4 years of training. This item analysis revealed that for the majority of items, the full range in the scales were used. Lastly, we used a single global readiness score to assess the correlation between physician and staff assessors’ readiness scores for the specific physicians they were evaluating. Because the data were not normally distributed, we used the nonparametric Spearman rank correlation coefficient to conduct this analysis (SPSS version 29 [IBM]). Ultimately, we plan to compare each item as well as summary scores when we publish outcome data that compares 3 versus 4 years of training from the LoTP and will generate separate scores for the physician and staff assessors.

Frequencies from the 2016 physician assessor survey are shown in Table 3. Scores in the highest performance category (“practicing Independently, rarely requests assistance”) ranged from a low of 32.8% for managing inpatient care, discharge planning, and transitions of care to a high of 93.8% for providing preventive care that improves wellness, modifies risk factors for illness/injury, and detects illness in early treatable stages. Note that several variables, especially those related to maternity care (48.4% prenatal care, 62.6% manage labor/delivery) and end-of-life care (57.8%) were not done in practice and could not be assessed. The lowest performance category (“not practicing very independently/frequently needing assistance”) was used for only one variable, “provide leadership within an interprofessional team,” in the assessment of just two individuals. Scores in the middle category (“practicing mostly independently, sometimes requests assistance") ranged from 1.6% to 20.3%, indicating a satisfactory range for assessment.

Frequencies from the staff assessor survey are shown in Table 4. The range in scores for interprofessional communication variables for the highest category (“always”) was 32.3% for “apologizing to you for inappropriate behavior” to 95.2% for “shows respect for you as a team member.” The category least used by the assessors was “rarely” for interpersonal communication variables, with “never,” “sometimes,” and “frequently” used more often. The range in scores for the processes of care variables for the highest category (“among the best”) was 41.9% for “handles transfers of care effectively” to 67.7% for “is courteous to coworkers.” None of the staff assessors used the “among the worst category,” while “below average,” “average,” and “above average” were used more often.

Fifty-four out of 64 surveys (84.4%) could be included in the assessment of correlations between the two instruments. The mean summary score for physician assessors was 14.37 (SD=0.98), and the mean for the staff assessors was 18.35 (SD=3.03). The Spearman rank correlation coefficient was 0.107 with a P value of.22, indicating no significant correlations existed between the two different assessors (data not shown).

 We undertook a particularly rigorous process in designing and pilot testing these two surveys to assess clinical preparedness for independent practice. We already had developed a graduate survey, which was administered 1 year after LoTP residents graduated from training. Because we wanted more than one metric measuring clinical preparedness, we developed surveys designed to be completed by both physician and practice staff assessors 3 months after graduates started their first posttraining position. We considered this timing optimal because graduates have become familiar with the logistics related to providing care in a new setting but may not yet have started on-the-job clinical learning. In addition, the 3-month window provides physician and staff assessors adequate time to assess the graduate’s preparation. We do not know, however, whether this time interval is valid because a similar tool does not exist. We used a consensus process with key stakeholders to come to agreement on this interval, so it was not chosen arbitrarily, but we do not know whether ratings of clinical preparedness would have been better or worse if a different time interval was chosen. In addition, our correlational assessment revealed no statistical correlation between physician and staff assessor scores. We believe that this occurred because the assessors were evaluating different measures. The physician assessors were evaluating clinical care, which staff assessors do not have the training to do accurately. The staff assessors were assessing processes of care performance, so we did not believe they would necessarily be correlated. As a result, we planned to present outcome data separately for each assessor.

 Additional issues related to validity and reliability are important to discuss. The EPAs that form the basis of the physician assessment were developed by leaders in the discipline, thus have face and content validity. These are used to assess resident progression during training, and we are unaware that other validity testing has been done with them. The items on process of care that we included on the staff assessor survey have been validated previously, and we plan to recheck these when we publish our outcome data. Because a study like this has not been done before, we do not have other available instruments to compare our findings to for additional validation. We understand the challenges of rater variation, which is why we added the threshold setting scenarios to each survey. Lastly, because assessing physician performance is dynamic and not a static phenomenon, conducting test-retest approaches would not be a valid approach for testing. We are hoping that these instruments may be used in other studies, which would allow for future comparisons.

Multifocal assessments would be beneficial, but measuring performance is complex. For example, measuring efficiency using time spent on the electronic health record may be affected by patient complexity, as can late chart completion and the number of consultations a physician makes. Assessments of diagnostic errors also can involve a series of events and several individuals and are not free of measurement error. The inclusion of self, patient, and supervisor assessments is subjective; and while knowledge assessments have been well-validated, they may not illustrate the application of knowledge. Nevertheless, as physician training moves closer to being competency-based, discussion of valid, reliable, and effective measurement should and will continue.

Decisions related to length of training in family medicine residency are important. Leaders in the discipline as well as residency directors have strong, and often diverging, opinions. ³¹ We intentionally involved many key stakeholders in the Measurement Summit and in the initial key informant interviews. Lastly, the survey pilot testing period was critical to creating a widely accepted and robust measure. A key question often raised is whether a 4-year graduate performs any differently or competently than a 3-year graduate after 1 year in practice. A description of the development and testing of these surveys is important to present in a freestanding paper; so when findings from these surveys are published, this paper may be cited as a reference. To our knowledge, we are the first to develop and rigorously test surveys designed to be completed by clinical practice assessors. Most studies on clinical preparedness survey either the residents themselves or their program directors. ^6-10 We are hopeful that this work will allow us to assess differences in clinical preparedness in their first practice after residency among residents who underwent 3 years of training compared to 4. We purposefully delayed reporting on survey development until data capture on clinical preparedness in 3- and 4-year residency programs was completed because we wanted to avoid any influence publishing the survey might have on physician and staff assessors.

The LoTP is, by intention, a pilot study. As such, it is not fully powered to test its hypotheses but rather to explore them with an eye toward identifying effect sizes that may inform larger studies with more rigorous study designs than the case-control design in place for the pilot. The assessment of survey frequencies indicated an appropriate range in responses, such that when generating a summary performance score, will likely be able to discriminate readiness for independent clinical practice according to length of training. We did change one response category in the survey for the variable “apologizing to you for inappropriate behavior,” where we dropped “rarely” because it could indicate either that they rarely apologize or that a need to apologize is rare because no inappropriate behavior is observed. This is the only revision we made to the survey.

The strengths of our approach include the involvement of a diverse set of leaders in family medicine, evaluation experts, and other stakeholders from rural and urban communities as well as from academic and nonacademic clinical settings in the development of our clinical preparedness measure. Rigorous pilot testing resulted in a feasible and understandable survey that generated a satisfactory range of assessor ratings. Getting this initial input will provide confidence in both the interpretation of findings from LoTP and how these findings can be used to advance our understanding for family medicine residency training.

The Length of Training Pilot is sponsored by the Accreditation Council for Graduate Medical Education and is funded by the American Board of Family Medicine Foundation.