The Impact of Length of Training on Clinical Preparedness Among New Graduates: A Report From the Length of Training Pilot Study in Family Medicine

The optimal length of training in family medicine, which varies globally between 2 and 5 years,¹ has been debated for decades.^2- 5 Some argue that additional training is needed to ensure preparedness for independent practice given duty hour restrictions, expanding curriculum needs, and evolving clinical and payment models for primary care practice. Others argue that the current length (2 years in Canada and 3 years in the United States) is adequate, and residency curriculum could be revised to address emerging issues. These debates have lacked rigorous data to inform them. Current assessments of clinical preparedness have been based on resident self-reporting or program directors’ evaluations prior to graduation.⁶, ⁷ Such approaches can be affected by recall and social response bias. For example, program directors may want to avoid being perceived as not preparing graduating residents fully, and residents may inflate or deflate their assessment of themselves. ^{8, 9}

To explore these issues, the Length of Training Pilot (LoTP) in family medicine was funded by the American Board of Family Medicine Foundation in 2013. ¹⁰ One of the study’s core questions was designed to explore the extent to which training length affected preparedness for independent clinical practice. We sought to overcome weaknesses in prior preparedness assessments by developing new survey instruments designed to be completed by two independent observers of new graduates: a supervising physician and a staff member in the graduate’s new practice.¹¹ A key feature of our assessment was the extent to which new graduates requested assistance as a marker of their preparedness for independent practice. The relationship between requesting assistance/supervision and progressive independence is well established in clinical psychology and medical education literature.^12-16 In fact, consequences have been identified when progressive independence is not well-instituted or erodes during training.¹² The purpose of this paper is to report findings on clinical preparedness of graduates with 3 versus 4 years of training in the LoTP.

The LoTP was designed to explore the impact that length of training, 3-year (3YR) versus 4-year (4YR), has on several program and learner outcomes in family medicine. ¹⁰ Briefly, it is a mixed methods prospective case-control pilot study including 17 residencies that applied for and were selected to participate in the study (seven 3YR civilian programs, six 4YR civilian programs, four 4YR Navy programs). The training models varied with four 4YR programs having a required fourth year with an integrated curriculum and two 4YR programs having an optional fourth year. The Navy programs were excluded because predetermined military objectives differ from civilian programs. All evaluation activities were overseen by researchers in the Department of Family Medicine at Oregon Health & Science University (OHSU). All LoTP programs obtained Institutional Review Board approval, and the evaluation team at OHSU was granted an educational exemption (IRB # 9770). Prior published papers describe other findings derived from the LoTP. ^11-19

A prior publication details the methods undertaken to develop and test the surveys used in this study.¹¹ Briefly, we performed a literature review of assessments for independent clinical practice, conducted key informant interviews with nine rural and urban family physicians regarding how best to assess recent graduates of family medicine residencies, and then convened leaders in family medicine, expert evaluators, and other stakeholders to take part in a Clinical Preparedness Measurement Summit in 2015 to guide decisions related to measuring preparedness for independent clinical practice.

At the Summit meeting, we defined clinical preparedness as “the extent to which graduates of family medicine residency training are independent/self-reliant in practicing core skills in the care of patients.” We also determined that the settings in which the care was provided needed to be comprehensive, including outpatient, inpatient, and other care settings (eg, home, nursing homes). We used the 2015 Entrustable Professional Activities (EPAs) for Family Medicine End of Residency Training as the framework for assessment of clinical preparedness by physician assessors. ²⁰ For the instrument designed for staff assessors, we identified nine validated variables in published literature that assessed interpersonal communication skills ²¹ and seven validated variables designed to measure processes of care delivery. ²²

The final draft surveys contained 61 items for the physician assessor and 36 items for the staff assessor. Two question sets on the physician assessor survey included scaled items. The first set assessed preparedness issues that we identified with our key informant surveys (eg, confidence and timing). This scale included three responses (1=no challenges, 2=some challenges, 3=many challenges). The second set focused on the EPAs and used the following scale: 1=not practicing very independently, frequently requests assistance; 2=practicing mostly independently, sometimes requests assistance; 3=practicing independently, rarely requests assistance.

Prior to administration, the question sets were extensively pilot tested using cognitive interviewing techniques ²³. Both surveys are available for review in our previously published paper.¹¹ Neither of these surveys collected qualitative data; thus, only quantitative data are presented here. We assessed graduates after being oriented to their new position but before significant on-the-job learning could occur (eg, 3 months after the graduates started their first position as a clinician).

Author A.E. worked with residency program coordinators to determine where graduating residents were entering practice after completion of training. She then contacted these practices to identify the graduates’ start dates, along with the best physician and staff assessors to receive surveys after new graduates had been on the job for 3 months. We asked participants to complete the surveys within 2 weeks, and up to three reminders were sent at 2-week intervals. Data collection started in 2016 and continued through 2023, with an overall response rate of 42.1% for the physician assessor survey and 39.0% for the staff assessor survey.

As in other LoTP analyses, we found that some graduates of required 4YR programs completed only 36 months of training because they graduated before the 4-year curriculum was fully implemented. Also, the optional 4YR programs had residents who graduated after 36 months of training. To accommodate these differences, we used both an intention-to-treat analysis²⁴ or program level analysis and an as-treated analysis,²⁵ leading to three groups for analysis: (a) graduates of 3YR programs; (b) all graduates of 4YR programs with either 36 or 48 months (4YR–36/48); and (c) graduates of 4YR programs with 48 months of training (4YR–48).

We used descriptive statistics to characterize graduates’ and assessors’ demographic information by length of training group, including mean and standard deviation as well as frequencies and percentages, and we summarized physician and staff assessor demographic information. To assess differences between groups, we used the Fisher exact test or χ² test for categorical variables and the independent samples t test with unequal variances for continuous variables.

Frequencies and percentages of assessment for preparedness by physician and staff were summarized according to each response category. For response options “not done in practice” and “cannot assess at this time,” we considered these a single category when deriving P values. Missing variables were excluded from analyses.

P values were derived from χ² tests, testing differences between 3YR and 4YR–36/48 groups and between 3YR and 4YR–48 groups. To determine effect sizes beyond statistical significance, we calculated φ coefficients for dichotomous outcomes and Cramer’s V for outcomes greater than two levels. φ coefficients typically range from 0 to 1 with an effect size of about 0.30 considered medium or meaningful.²⁶ Cramer’s V is calculated based on the degrees of freedom used in analyses (based on the number of rows and columns). For our study, a meaningful effect size (medium) ranged between .15 and .21.²⁷ Effect sizes with negative values indicate a reverse relationship. We performed analyses using R software version 4.3.0 (R Foundation). All tests were two-tailed, and we set α at 0.05.

The study population included 403 graduates of 3YR programs and 459 graduates of 4YR programs (36- and 48-month programs combined). Of these, 274 (59.7%) received 48 months of training (Table 1). The mean age across study groups ranged from 33.1 years (SD=3.9) to 33.7 (SD=3.7). The majority of graduates were non-Hispanic White (64.3% to 70.2%), US medical school graduates (70.5% to 85.4%), who had not enrolled in or completed a fellowship (65.7% to 75.2%); though more graduates in 4YR programs reported having completed or were in fellowship training (P<.006).

Characteristics of physician and staff assessors were similar across study groups with one exception (Table 2, Table 3 ). Staff assessors of 3YR graduates tended to be medical assistants, while assessors of both 4YR–36/48 and 4YR–48 graduates tended to be registered nurses (50.8% vs 39.5% to 39.8%; P<.004).

Physician assessors indicated that they typically worked 7 to 10 half days per week and that the information sources they used to assess graduates included direct observations (81.7% to 86.5%) and interactions with physicians (82.4% to 85.8%; Appendix Table A). The majority of physician assessors indicated that they consulted with new graduates frequently (69.0% to 70.5%). For 3YR program graduates, 82.2% of staff indicated that they worked every day or nearly every day with their physician colleague, compared to 68.8% in the 4YR–48 group. Across all groups, direct observation and interactions with the physician were the most common sources used by staff in their assessments. Among those that reported practice setting, academic health centers (14.8% to 18.4%), federally qualified health centers (18.4% to 22.3%), and hospital/health system-owned medical practices (9.8% to 16.9%) were all represented. Cramer’s V yielded a meaningful effect size for 3YR graduates compared to 4YR–36/48 and 4YR–48 graduates’ comparisons of “How often do you work with this physician while caring for patients?”

Physician assessor ratings of EPAs were similar across study groups except for three items (Appendix Table B). First, for providing a usual source of comprehensive, longitudinal medical care for people of all ages, 86.5% of 4YR–48 graduates were “practicing independently” compared to 77.9% for 3YR program graduates (P=.036). Second, for managing prenatal care, 41.2% of 3YR graduates were “practicing independently” versus 63.1% for 4YR–48 graduates (P=.007). Of note is that 40.4% of 3YR program graduates did not provide prenatal care in practice, compared to 21.6% of 4YR–48 graduates. For managing labor, delivery, and postpartum care, both the 4YR–48 and 4YR–36/48 groups had a higher percentage of graduates “practicing independently” (4YR–36/48, 38.6%, P=.014; 4YR–48, 41.4%, P=.005) compared to 25.7% for 3YR graduates. Again, of note is that 54.4% of graduates of 3YR programs don’t provide maternity care in their practice, compared to 37.8% among graduates in the 4YR–48 group. Third, when assessing specific care processes, physician assessors were more likely to rate 4YR–48 graduates as having “no challenges” with speed/timing related health care visits (91.9%) compared to 3YR graduates (82.4%, P=.030). Effect sizes yielded from Cramer’s V for all significant P values were meaningful effect sizes.

Findings from staff assessors are shown in Appendix Table C. The only interpersonal communication characteristic noted to be different among the groups was in explaining the rationale for their care plans or actions with a rating of “always” for 82.2% of 3YR graduates compared to 70.1% or 68.9% of 4YR–36/48 or 4YR–48 graduates (P=.026 and P=.016). Processes of care characteristics rated as “among the best” were similar across all study groups with a range of 45.7% to 71.9%. For staff assessments of challenges with care process areas, graduates rated as having “many challenges” ranged from 0% to 4.3% with no significant differences among the comparison groups. Cramer’s V effect sizes for all significant P values were meaningful.

In this study of the impact of length of training in family medicine, we found that for three of the 21 EPAs for residency graduates, those receiving 4 years of training were more likely to be rated by physician assessors as “practicing independently” 3 months into practice compared to graduates of 3YR programs. This was true for the 4YR–36/48 and 4YR–48 groups, which suggests that something about these 4YR programs results in their graduates being better prepared. Graduates of 4YR programs were better prepared for EPAs of providing a usual source of comprehensive longitudinal medical care for people of all ages; managing prenatal care; and managing labor, delivery, and postpartum care. Possibly these graduates purposely chose a 4YR program because they wanted to provide a full scope of services to their patients. This finding is reinforced by the fact that the practice settings did not differ among graduates of 3YR and 4YR programs. Given the rising complexity of patient care and shortages of well-trained primary care clinicians, producing graduates better prepared is of value to physician employers and society as a whole.

We know that pregnancy care provided by family physicians is declining, ^{28, 29} which is less impactful in urban areas than in rural areas, which have fewer, if any, obstetricians. Four-year residency programs are resisting this trend by offering more robust pregnancy care curricula. These graduates are greatly needed, particularly in rural and underserved areas.

We also found that speed/timing related to visits was rated by physician assessors as having fewer challenges for graduates with 48 months of training. Perhaps the additional year of training, which a prior LoTP paper found to result in approximately 1,000 more patient visits at graduation, ¹⁸ better prepared these graduates, compared to graduates of 3YR programs, to work more effectively from the beginning, perhaps by integrating complex medical issues into a single visit. Interestingly, we found no differences in staff members’ assessments of new graduates according to length of training. Staff assessments focused on interpersonal behaviors and communication skills rather than assessments of clinical care, which may indicate that these areas of practice are more sufficiently developed after 3 years of training.

We found that many variables did not differ between graduates of 3YR and 4YR programs, which suggests that many areas of practice can be well-developed after 3 years of training. Caring for patients with complex and chronic medical and mental issues in multiple settings appeared not to differ across study groups; similarly, we found no differences in managing acute illnesses, performing common procedures, managing end-of-life care, using data and best science to optimize and coordinate care, and providing leadership within an interprofessional team. These last variables represent more recently emerged competencies for family physicians; 3 years of training appears to prepare graduates well in these areas.

Of note is that graduates of 4YR programs reported having undertaken fellowship training. We suspect that these respondents were considering their fourth year as a fellowship year. In addition, graduates of 3YR programs were more likely to have been assessed by a medical assistant rather than a nurse, which more likely occurred among 4YR graduates. This finding suggests a potential difference in practice staffing that affects which practices graduates choose to join post training.

These findings are important given the novel feature of assessing new graduates after being in their practice for only 3 months—an interval we felt would best reflect their readiness for independent practice as a result of their residency training rather than the education that occurred on the job. One of the final papers to emerge from the LoTP will address self-reported scope of practice, assessed 1 year after graduation, and will shed additional light on the impact length of training has on practice.

Strengths of this study included the effort that went into developing and testing the clinical preparedness instruments used and the inclusion of programs from geographically diverse areas and settings. Weaknesses included the less than desirable response rates of around 40%, which are not generalizable to the study groups and suggest that we could likely have biases, including recall and social response biases. Though the literature has suggested that progressive independence typically includes self-reliance, ^{15, 16} some personality characteristics may have influenced new graduates’ requests for assistance, though we have no reason to believe this would differ across study groups. Differentiation between the study groups could become more apparent with a larger sample.

The 4YR programs were selected based on interest in the LoTP and their ability to implement a 4-year curriculum. So, the programs that produced these cohorts of graduates may not be fully comparable. Another weakness is that this study was designed as a pilot and was not fully powered to conduct a rigorous assessment of hypotheses, though it did provide invaluable information about effect sizes across our study groups. We did find some differences according to who the staff assessor was (eg, registered nurse vs medical assistant), which may have caused some measurement errors. We also choose not to account for multiple comparisons to avoid Type 1 errors because that would have resulted in us setting the α level at P<.0004. Doing that would have increased the likelihood of Type 2 errors (making false negative conclusions).

The EPAs that provided a framework for our survey were developed in 2015; since then, in 2022, the Accreditation Council for Graduate Medical Education Family Medicine Review Committee and the American Board of Family Medicine (ABFM) developed Core Outcomes consisting of 12 competencies measuring the foundational knowledge, skills, and activities needed at the beginning of independent practice. ³⁰ The EPAs did not seem to find a place in residencies, with a minority of programs using them as an assessment tool.³⁰ Unfortunately, the Core Outcomes were not available when we were developing our instruments. Future research on clinical preparedness should use the ABFM’s 12 competencies, and ideally the new ACGME AIRE program in family medicine will continue researching length of training. That program allows residencies to pursue innovation and offers freedom from program requirements with ongoing assessment of outcomes, as long as the residencies have approval from the ACGME specialty’s Review Committee and ABFM. ³¹

In conclusion, the differences we found related to a longer length of training appear to be primarily related to comprehensive, longitudinal relationships with patients; pregnancy care; and speed and timing in providing patient care, which were more discernable by the physician assessors than the staff assessors. Most competency variables did not differ according to length of training.

The Length of Training Pilot is sponsored by the Accreditation Council for Graduate Medical Education and is funded by the American Board of Family Medicine Foundation.