Simulation-Based Medical Education in Family Medicine Residencies: A CERA Study

To prepare family medicine residents for the increased spectrum of clinical skills they must acquire within the constraints of reduced duty hours, limited patient encounters, and increased focus on patient safety, simulation education has been proposed as a method to expose residents to common and uncommon clinical situations.1 Simulation-based medical education (SBME) leads to retention of clinical knowledge 2 and improved performance in rare, high-stakes situations,3,4 and exposes residents to clinical scenarios of varying difficulty.5 SBME can improve a resident’s self-confidence,6 medical knowledge, clinical skills, communication,7-9 critical thinking,10 team building, and leadership11 —all items evaluated on resident Accreditation Council for Graduate Medical Education (ACGME) milestone evaluations.12 When theoretically-grounded, SBME can help residents achieve more specific clinical skill acquisition goals as compared to traditional clinical medical education.13 SBME is also an effective tool for gathering professional assessment feedback14 and for the evaluation of clinical skills in residency training environments.15,16

Though simulation has been a staple of graduate medical education (GME) for more than 40 years in the fields of surgery, anesthesiology, and emergency medicine, family medicine (FM) historically has not fully leveraged the potential of simulation. A 2011 AAMC survey regarding simulation showed that only 12% of teaching hospitals and 58% of medical schools utilized simulation for family medicine resident education.17 Potential barriers to the use of simulation in FMR programs include lack of fiscal and physical resources, such as simulation models or simulated patient actors, and lack of faculty simulation experience.10

Since the AAMC study in 2011, no studies have investigated the scope of SBME in family medicine residencies. This study aims to fill that gap. The objectives of this study were (1) to describe the current use of SBME in FMR education, (2) to identify FMR program characteristics associated with the use of simulation, and (3) to assess the relationships among program director (PD)-perceived barriers, PD-perceived value, and the use of SBME.

To assess the current state of simulation education in family medicine and to better understand the relationships between program characteristics, resources, and PD characteristics and the use of and barriers to simulation use, we developed questions for inclusion in the 2019 Council of Academic Family Medicine (CAFM) Educational Research Alliance (CERA) survey of FMR program directors. The methodology of the CERA Program Director Survey has previously been described in detail.18 The American Academy of Family Physicians Institutional Review Board approved this project in September 2019.

The population surveyed was all ACGME-accredited US FMR PDs as identified by the Association of Family Medicine Residency Directors (AFMRD). Email invitations to participate were delivered along with the survey, using the online program SurveyMonkey. Between September and November 2019, six follow-up emails were sent after the initial invitation to encourage nonrespondents to participate. At the time of the survey, 668 PDs met the inclusion criteria. Forty had previously opted out or blocked surveys; therefore, the survey was emailed to 628 individuals.

The research team developed survey questions to assess the current state of simulation and submitted the questions through the CERA survey process. The CERA Steering Committee, along with a group of expert peer reviewers, evaluated questions for consistency with the overall subproject aim, readability, and existing evidence of reliability and validity. Pretesting, conducted with family medicine educators who were not included in the sampling frame, evaluated questions for flow, timing, and readability. The final larger omnibus survey included 53 survey items: 10 demographic/characteristic core items and five topic-specific sections. The simulation section included nine questions.

Main variables of interest included use of simulation and perceived barriers to simulation use in resident education, assessment, and remediation. We provided the following definition of simulation to respondents: “a method used in health care education to replace or amplify real patient experiences with scenarios using lifelike mannequins, physical models, standardized patients, or computers.” Likert-type items assessed the value of the use of simulation in education, assessment, and remediation. Four questions addressed the specific uses, how often simulation was used, and which competencies simulations assessed. Four questions addressed the types of simulations used to create realistic simulation experiences. Table 2 provides definitions of each of these simulation types. PDs were also asked to identify barriers to the use of simulation from a pre-defined list developed from previous literature and to identify their access to dedicated simulation location, defined as “a space or building that has a primary role/purpose to provide simulation-based medical education.”

In the core questions, PDs reported their residency program’s general characteristics (location and size of program, size of community, proportion of international medical graduates (IMGs) and individual characteristics (PD gender, race, and length of time in PD role).

We coded, aggregated, and analyzed data using Statistical Analysis System (SAS) software version 9.4. We compared differences in simulation use, how often simulation is utilized, how programs utilize simulation, competencies, barriers, and value of simulation between type of residency program and size of residency program using a χ2 test of independence and/or Fisher’s exact test as appropriate. In the direct comparison between community-based, university-affiliated, university nonaffiliated, and university-based FMR programs, “military” and “other” residency program types were excluded (n=10), due to not being able to confidently distinguish them as community-based or university-based programs. In direct comparisons between community-based and university-based FMR programs, community-based programs were defined to include both university and nonuniversity-affiliated community programs. We used a Spearman rank test post hoc to explore the association between value of simulation for education and assessment and frequency of simulation utilization.

The overall response rate for the survey was 39.1% (250/628). Table 1 presents program characteristics of responding PDs.

Overall, 84.5% of respondents reported that their programs use simulation. A majority of responding PDs (67%) report using simulation for both education and assessment. The core competency that was assessed the most through simulation was patient care, followed by medical knowledge and interpersonal communication skills (Table 2). PDs reported that they use a variety of methods of simulation, including standardized patients (56%), full-scale mannequins (61%), and part or partial task trainers (59%). Screen-based simulation was utilized the least; only 22% of PDs reported use of this method of simulation (Table 3).

A significant, positive linear relationship existed between perceived value of simulation for education and assessment and frequency of simulation utilization. The more frequently simulation was utilized in a program, the more a PD valued simulation as a technique for resident education, including remediation (s_r=0.35, P<.0001), and assessment (s_r=0.22, P=.0014).

Residency programs of different types and sizes utilized SBME at comparable rates. Similarly, there were no significant differences between university- and community-based programs regarding how simulation is used, how often it is used, and which of the ACGME core competencies are assessed using SBME (Table 2). Access to a dedicated location for simulation did not differ significantly between sizes of family medicine programs (79% and 77%). Slightly more university-based programs utilized simulation for education, including remediation, compared to community-based programs (37% vs 21%), although this difference was not significant.

PDs of community-based programs (18%) were as likely to report barriers as PDs of university-based programs (23%), although the most prominent barrier differed significantly P=.012). For university-based programs, the most prominent reported barrier was lack of faculty teaching time. However, for community-based programs, the most prominent reported barrier was lack of support from teaching hospital or academic institution.

Fifteen percent of PD respondents reported that their program did not utilize simulation. Participants who did not utilize simulation reported significantly lower rates of agreement with the statement “simulation is a valuable technique for education” (71% vs 92%, P=.001) and “… for assessment” (66% vs 82%, P=.0286) compared to PDs that utilized simulation, respectively (Table 4). Of PDs who did not utilize simulation, 61% reported not having access to a dedicated location for simulation compared to 18% of programs that utilized simulation (P<.0001). Among PDs who did not utilize simulation, 95% reported a barrier (ie, lack of resident and/or faculty time, interest, or training; lack of support from teaching institution or colleagues) when attempting to provide simulation-based education compared to 80% of programs that utilized simulation (P=.034).

In this 2019 CERA survey, a majority of responding FMR PDs in this survey reported that they utilized simulation as an educational tool, for both education and assessment. PDs also reported access to dedicated locations to utilize SBME. Most FMR programs using simulation utilize it for clinical skills and medical knowledge core competencies. Although studies in other fields have highlighted the value of simulation in improving professionalism,19,20 executive leadership,21 and skills in systems-based practices,22 such as addressing patient safety,21, 23 FMR PDs here do not report using SBME to affect these competencies. In undergraduate medical education, simulation is also successfully being used to teach interprofessional teamwork.24

These results suggest that FMRs are not leveraging SBME to its greatest potential. In this sample, PD perceptions of value were related to the use of simulation. This may indicate one potential target for intervention. FMR educators may not be aware of these possible applications or may not be trained to adapt simulation to include these competencies. More faculty development may be needed to share the evidence on the positive effects of SBME.

A majority of respondents, even those utilizing SBME, reported barriers to simulation. A lack of support from sponsoring institutions can be related to lack of investment in simulation infrastructure, changing educational culture, or training faculty. In this study, residency programs with characteristics that are associated with more resources, such as larger size and university affiliation, were just as likely as their counterparts to report barriers to simulation use. The top-three barriers reported from FMR programs include lack of resident and faculty time, lack of support from teaching hospitals, and lack of faculty training. Effective SBME requires time investment for proper coordination, development, and execution.1 Time and resources are also needed for organized faculty development, including training faculty in SBME-specific skills such as debriefing. 25 Organizational leaders can enable implementation of SBME by resourcing spaces, equipment, and staff. 26

Findings are limited by the self-selection of respondents to participate. Although the response rate was below the recommended 60% threshold,27 one strength is that respondents represent a wide distribution of residency types and sizes. Respondents may not have had a shared mental model of the difference in simulation for assessment versus for education, which could limit application of findings. It is possible that the respondent group was systematically more or less interested in simulation than non-respondents; however, the simulation questions were among others about unrelated subjects in family medicine education, lessening this concern. This study was also conducted just before the release of a new milestone evaluation system by the ACGME in July 2020,28 which may subjectively change responders’ interpretation of their use of simulation to evaluate milestones.

Future intervention and implementation studies are needed to identify how to overcome barriers and how to make SBME more sustainable and effective for FMR programs. Data on patient safety and interprofessional team care would greatly improve the understanding of how SBME can be effective.