Residency Education Redesign: The Interplay of Innovation and Standardization

Family medicine (FM) was created in 1969 as an innovative response to decreasing numbers of general practitioners, increasing specialization in medicine, and the need for whole-person and family care.1 Over the years, with strong leadership from residencies, FM has continued innovating in response to community needs by adopting new therapies, such as medication-assisted treatment (MAT) and transgender health care.2,3 The COVID-19 pandemic further underscores the need for flexibility, as family physicians quickly transitioned to virtual care, prepared for inpatient surges, and led public health initiatives.4

FM residency training requirements have expanded considerably from 4 to over 61 pages,5 as have innovations in medical education, including graduate medical education milestones6 and entrustable professional activities (EPAs).7 While these advances are theoretically compelling, tensions have developed regarding how best to balance innovation and standardization. These tensions generate confusion among FM’s specialty colleagues, employers, and the public regarding what to expect from residency graduates in practice scope and expertise across all patient populations (eg, maternity and pediatric care).8

Recently, the Accreditation Council of Graduate Medical Education (ACGME) fostered innovation in a number of ways, including identifying “detail” requirements that describe processes that programs may use or not. This latitude permits successful programs to innovate in certain areas.9 Over the last decade, much has been learned from several collaborative efforts sponsored by the ACGME, FM, and other primary care specialties.10-12 Yet, it is unclear how standardization fits into this balance.

The aim of this paper is to provide an analysis of how standardization, especially of residency outcomes, and innovation relative to educational structures, methods, measures and requirements have helped to advance as well as create complexities in FM, which will help chart a path into the future. We believe both are needed and are interdependent—innovation should only proceed when standardization of milestones and frameworks as well as of the outcomes of residency training have been clearly identified so they can contribute to the next iteration of standards. The outcomes of residency education are especially crucial because they contribute so importantly to accountability. If new residency requirements will soon emerge, now is the time to be very explicit about requirements that take advantage of the educational sciences, so residencies can redesign training toward producing an ever-improving physician prepared for the big future changes and unmet needs of their patients. By this, we do not mean to suggest that educational research should replace clinical research or quality improvement. Both are needed. Our analysis will proceed through four phases: (1) The Why, (2) Possible Options, (3) What the Evidence Shows, and (4) Recommendations.

Innovation in medical education allows for the study of improved educational approaches that contribute to the science of learning.13 Benefits of innovation include that programs may be highly motivated to try new approaches that can help FM improve and grow. Many residents in innovative programs are freer to explore areas of interest and develop individualized curriculum.¹² Innovation can also drive collaboration among programs, even those that have historically viewed each other as competitors.14 Innovations inherently involve some risk, as the associated outcomes, both clinically and educationally, are yet to be fully determined.

Benefits of standardized requirements include that they are known, which adds stability. Successful programs using the same curriculum would expect educational outcomes to be similar, if not the same. Residents, employers, and sponsoring institutions have a clear understanding of what to expect of respective programs’ graduates. Challenges of standardization include that it can lead to program stasis with failure or delays in adapting to a changing environment. It may also drive training to resource-rich areas where meeting requirements is easier, while decreasing training in underresourced settings serving disadvantaged populations. Standardization also supports an underlying sense that training is driven by ACGME accreditation rather than patient care and communities served; however, it can also impede innovation.

In 1999, the ACGME approved six core competencies applicable to all disciplines, after which residency and fellowship programs were expected to use this framework to improve curricula and assessments.¹⁵ Yet, implementation proved difficult as programs struggled with assessments of professionalism, practice-based learning and improvement, and systems-based practice.16 In addition, research illustrating that standardized graduate medical education requirements produces standardized results, especially for patient outcomes, has been lacking.17 These challenges led to AGCME’s 2010 Milestones project.18

The Milestones represent a complex interaction between educational intervention and assessment, both of which are now part of accreditation. Though early assessments are contributing to Milestones validity,19 more work must be done to fully realize the Milestones’ impact. Shortly after ACGME Milestones development, EPAs emerged as a concept that allows faculty to make competency-based decisions regarding the level of supervision required by trainees.7 Some disciplines, as in the case of the American Board of Pediatrics (ABP), have evaluated the integration of assessment frameworks for determining trainee readiness for independent practice. The ABP now includes core competencies along with both milestones and EPAs because their own research has shown that these fill gaps in assessment that can result when either is used alone.²⁰

The current state of education and assessment for accreditation is more integrated than ever before. The competencies, subcompetencies, and milestones developed via the ACGME process are crucial for assessing residents as they progress through their training and represent further standardization.21 With these measures firmly in place, programs can demonstrate success or failure of innovations tied to explicit outcomes rather than being based on faculty and resident perceptions. Nevertheless, clinical time and residency administrative work can be overwhelming, and staying up to date with existing literature on Milestones and EPAs is challenging. It may be that the balance between standardization and innovation will depend on faculty development; however, what this should include is uncertain. At the specialty level, the role of collaboratives has been shown to be important for implementing and assessing innovation in meaningful ways.21 Methods for increasing program involvement in such collaboratives are untested at the program level.

What possible options would advance the balance between innovation and standardization? We believe the following are likely to be most impactful: (1) expanding and studying the scope of requirements, (2) adding skill-based rather than time-based faculty requirements, (3) further altering scholarly activities requirements, and (4) requiring participation in multisite collaboratives that include rigorous assessment designs.

Current graduate medical education requirements focus primarily on resident experiences, as measured by episodes of care/procedures or time spent in particular settings.⁵ As mentioned earlier, competence is primarily assessed through the six ACGME core competencies,¹⁵ and FM has 19 subcompetencies and associated milestones. At present, the evidence supporting these is weak.²² Preparing residents for independent practice may mean expanding the scope of these requirements to include EPAs, as the American Board of Pediatrics has done. As part of the Family Medicine for America’s Health initiative, a task force was created to develop a list of 20 EPAs for the end of family medicine residency training.²³ Ongoing studies of both milestones and EPAs will continue to be needed to advance their development and generate the evidence needed to guide their application.

The benefit of adding EPAs includes the clearer descriptions of what it means to do the work of a family physician. Subcompetencies and milestones describe some of the skills needed but don’t describe whether a resident can combine all patient care skills together and were never meant to be assessment tools.7, 24 In addition, the concept of entrustment is more clearly understood by faculty members or specialists working with residents. By making EPA entrustment decisions, faculty are implicitly making competency decisions on subcompetencies and milestones.25 Though actionable EPAs have been described by pediatrics26 and Canadian family physicians,27 not all of FM’s current EPAs are as actionable as they need to be in the United States. Revised EPAs that are built into the current competency-based assessment framework, rather than created separately, could facilitate curriculum refinement as well as resident evaluation. The benefit would be that the work of family physicians’ and residents’ development toward that work are well described and universally understood.28

Programs are required to designate core faculty members who are specifically tasked with devoting a significant portion of their overall full-time equivalent (FTE) to education, administration, teaching, evaluation, scholarship, and provision of formative feedback to residents. Currently, programs are required to have a core faculty, and the faculty development requirements are time-based rather than skill-based. As such, the assessments of faculty development programs should be much more robust and be based on continuous quality improvement principles to ensure continuous assessments result in a truly improved faculty. Core program faculty could benefit from training in leadership, change management, instructional design as well as program evaluation, and learner assessment. The National Institute for Program Director Development Fellowship program of the Association of Family Medicine Residency Directors covers several of these topics, but is not required of residency program directors. In addition, program directors could benefit more from additional training in educational sciences, such as attaining a master of science in education (MSEd), which would provide invaluable training in educational research designs, instrument development, testing, and mixed-methods approaches to analyses. If this is not possible, every residency should have someone with educational research training on the team, similar to the behavioral health requirement. At the very least, programs should have access to educational experts in instructional and assessment designs. This could occur through membership in a consortium where such expertise is available to enrolled members.

Initially, scholarly activities requirements were quite broad in FM, and as a result, residents and faculty sought opportunities elsewhere in their departments to meet these requirements, such as working with research faculty on their clinical areas of expertise. In July 2019, these requirements changed such that scholarly activities were to focus on the program rather than individual interests of faculty. While this is a positive change, more is needed to ensure that faculty and residents tackle topics that advance residency training. More research is needed on resident learning curve development and factors that may predict the need to expand the length of training according to learner development. For such research to be meaningful, there must be flexibility in accreditation standards, such as would occur if a prediction model was developed and validated that could show that additional training may be needed for residents who have certain characteristics. For example, flexibility should be allowed for those having children during residency in FM, a discipline that values families so highly.

Conducting innovative research in residency training is not generalizable if done in a single program. Thus, requirements could include offering ACGME innovation waivers as part of multisite vetted study protocols designed to answer specific questions in FM training. Examples of successful collaboratives include I3, P4, the Colorado Collaborative, and the Length of Training Pilot, which is still underway. Collectively, these efforts have involved important assessments of innovations that, in at least one case (I3) included patient outcomes. Partnering in this way would allow for more rigorous study designs, including randomized crossover or prospective case control designs. Establishing such an endeavor would exponentially advance educational science in graduate medical education. Additional requirements would include reporting outcomes back to the ACGME Review Committee for Family Medicine (RC-FM), presenting at national and international meetings and publishing in peer-reviewed journals.

Competency-based medical education (CBME) has been implemented across many specialties and countries over the past several years. The ACGME started this process first with the six core competencies and then with more specific subcompetencies and milestones with the adoption of the Next Accreditation System (NAS) in 2013.²⁹ At present, the evidence supporting these measures is weak, with one systematic review revealing a lack of evidence that these competencies can be independently measured.²² This systematic review also found that systems-based practice as well as practice-based learning and improvement are properties of systems rather than of individual trainees.²² As a program director, the opportunities to fully understand the concept of CBME includes discussion of EPAs,²⁷ which were foreign concepts at the time the NAS was proposed. How the components of CBME relate to one another was similarly unclear. The ACGME chose to recognize that EPAs could be useful but that the focus, initially, would be at the competency, subcompetency, and milestone levels.

Among the many assumptions made is that the skills of being a physician can be described adequately with competencies, subcompetencies, and milestones. Another is that the same CBME process can be as useful for complex interpersonal interactions (eg, family conferences, breaking bad news) as for technical skills, such as clinical procedures.30 As an educational model, most of the evidence generated has addressed the development of CBME components, but little has tested these assumptions.29 While the evidence supporting EPAs as an evaluation model is similarly slim, EPAs do have the advantage of describing complete physician tasks rather than just their components.31 In spite of the fact that the ACGME currently only requires milestone reporting, there is increasing interest in further work on developing EPAs.30

Faculty development is clearly important for physicians transitioning from solely providing patient care into the role of educator.32 This is especially true for assessment skills. A review of educational research in family medicine noted that studies tended to be quantitative, suggesting that faculty were not well trained in qualitative study design.33 These authors also found a large percentage of quantitative and qualitative studies for which no specific methodology was available.33 A more recent review showed that faculty development programs often do not undertake rigorous assessments of faculty educator outcomes.34 Emergency medicine has shown that, in addition to the complexity and scope required of faculty skills, attention to faculty development can and should include robust, measurable outcomes.35

Ideally, resident participation in scholarly activities would promote the practice of evidence-based medicine, critical thinking, quality patient care, and provide lifelong learning skills.³⁶ The ACGME requires scholarly activities of both residents and faculty,³⁶ but until 2019, these requirements were clinically focused, such as organized clinical discussion on rounds, journal clubs or understanding the basic principles on how to conduct research, evaluate its quality, explain it to patients and apply it to patient care.³⁶ In July 2019, the scholarly requirements for faculty changed by adding innovations in education to the list of scholarly activities. Missing from this list was scholarship in curriculum development and assessment, which would be necessary for educational development of residency training as well as the ongoing clinical development of master adaptive learners, whose development of self-regulation fosters the use of adaptive expertise in medicine.³⁷

Unlike evidence-based medicine, evidence-based or evidence-guided medical education has been slow to evolve. This may be due in part to the fact that studying a single program does not produce generalizable new knowledge, that learners perceive educational innovations as risky and shun participation, and lastly, that there is limited funding to support educational research.

Multisite educational research collaboratives have successfully fostered innovation while also including robust evaluations.^12,38 A multisite study conducted by the Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network²⁶ examined learning curves using EPAs among 1,987 pediatric residents. This is an excellent example of an impactful study in that the investigators found that at least 90% of trainees achieved levels of unsupervised practice for only eight of 17 EPAs required by the end of residency. This suggests that gaps exist between readiness for independent clinical practice and current practice standards for general pediatrics, a finding that needs to be addressed with further study.

Educational research collaboratives could foster research that can inform training and standards. Many initiatives have shown that residencies not only survive significant innovations, but also actually thrive while doing it.¹² A paper by Schwartz et al, published in 2016, identified 15 Medical education practice-based research networks across the United States.³⁹ Though several publications have shown that existing networks are not necessarily needed for successful collaboration,⁴⁰ innovation within collaboratives would expand the generalizability of findings, allow many programs to benefit, and assist with scholarly activities for both residents and core residency faculty.

The opportunity to revise residency requirements should include a change in their focus. Current requirements concentrate on resident development, training setting, and clinical learning environment. It is time to pivot to requirements that focus not just on current state, but also set a solid foundation for future excellence. Toward this end, we recommend that:

1. Adoption and study of clear and actionable EPAs as educational standards for residency graduates. Doing this will foster clearer communication to students, residents, faculty, employers and patients regarding exactly what it means to be a family physician. The EPAs must explicitly align with competencies, subcompetencies, and milestones that have already been developed. A clear definition of measurable outcomes is required for evaluation of any standards or innovation. Importantly, EPAs are the appropriate way to describe those skills which differentiate family physicians from other primary care providers.
2. Core faculty be required to engage in faculty development that includes training on competency-based medical education using the EPA framework, advanced curriculum development, program evaluation, objective learner assessments that align with individualized learning plans. Participation in formal faculty development programs must be required of all new core faculty members. We believe that to achieve this goal faculty will need to devote a minimum of 80 hours to this development, and that such programs should be required to include robust assessments and continuous improvement principles to ensure they are actually developing faculty skills. In addition, increased opportunities are needed to support program directors attainment of training in educational sciences.
3. Core faculty must have a minimum of 30% of their time available for program enhancements, administration, evaluation, resident assessment, and scholarly activities.
4. Require participation in educational collaboratives that rigorously study innovation. Examples of learning collaboratives could be geography-based, such as WWAMI (Washington, Wyoming, Alaska, Montana and Idaho Collaborative) or I3, or educational intervention-based initiatives, such as the Length of Training Pilot, Clinic First, or the American Medical Association’s Reimaging Residency initiative. Through these collaboratives, programs would have access to educational experts to assist with instructional design and evaluation. Each collaborative must be engaged in the study of at least one educational innovation using rigorous study designs and evaluation methods.
5. Scholarly activities of residents and faculty support the educational development of residency training via ongoing educational improvement and research projects as well as ongoing clinical development of residents as master adaptive learners.37

We believe both standardization and innovation are necessary for the discipline to advance. When programs innovate and evaluate their educational outcomes to find better ways to teach, this leads to future standardization based upon the evidence, and then further innovations can occur. Standards are required if innovation is to have meaning. Innovation is required if standards are to adapt to a changing healthcare landscape. These recommendations are not independent but describe a bold and active approach for continuous reinvigoration of the specialty. The specialty needs a clear statement of what family physicians do. EPAs are a way to achieve this without devolving into a list of conditions to be treated. If there are specific skills that set family physicians apart, we should be able to describe them in actionable ways, eg, what does it mean that a resident or family physician engages in continuity? Second, we need well-trained faculty members who can skillfully create and assess innovations in residency. Such innovations can include didactic curriculum, clinical setting, and the very structure of residency. Third, collaboration is essential if we are to understand the effects of innovation in any meaningful, reproducible way. Fourth, these robust results must then inform future standards for residency education requirements.

Virtually all of the above recommendations will have associated costs. Residency training has long been underresourced, and the federal GME fund flow lacks transparency and accountability.40 Being involved in innovation requires faculty time for development, implementation, as well as study/evaluation. The return on investment for the faculty time will be the evolution of residency training that ensures residency graduates are ready to provide the best in contemporary care. Many have called for revisions to GME training funding mechanisms, yet change has not occurred. It is infeasible to add residency requirements without providing resources needed to ensure they are met. These resources represent an investment in the future health of the US population. Residency training support should diversify in ways that engage primary stakeholders, including the federal government (Medicare/Health Resources and Services Administration), health systems/other employer groups and relevant foundations. All stakeholders in the discipline must be included in strategic/deliberate planning to address direct/indirect costs of residency training.

Implementing these recommendations, which represent a vital interplay of thoughtful innovation through broad collaboration toward evidence-guided standardization will help to ensure FM residency-trained graduates are ready to provide the best, most up-to-date care to their patients and communities now and into the future.