Comparative Outcomes of ChatGPT Versus Faculty-Facilitated Simulation Design by Medical Students

Interest is growing in applying generative AI models, such as ChatGPT, to simulation-based health care education, which is a mainstay in medical education for developing knowledge and skills in a safe learning environment.^1-4 Simulation-based manikin scenarios (SBMS) replicate various patient conditions, allowing learners to apply clinical knowledge and practice problem-solving, decision-making, and teamwork in a variety of specialties, across acute and primary care settings. Cocreating educational content with faculty guidance enhances learner engagement, promotes ownership of learning, and reinforces knowledge.⁵ It also allows faculty to improve teaching practices, gain insight into learner needs, and foster an inclusive learning environment.⁵

Existing studies examining ChatGPT’s role in SBMS design have focused on the use of AI by simulation experts or nonsubject matter experts (SME) demonstrating efficiency gains while identifying limitations in accuracy and content quality, but none have evaluated its application by novice simulationists.^4,6 Given their inexperience in scenario design, novice simulationists stand to benefit most from AI-assisted tools. However, unlike experts, they may lack the knowledge to identify inaccurate or suboptimal AI-generated content, making quality evaluation of AI-assisted scenario design particularly important in the population. This pilot study examined the ChatGPT-assisted design process and compared novice-designed ChatGPT-assisted generative AI (AIG) SBMS with faculty-facilitated medical student interest groups (MSIG) SBMS.

The University of Hawaii Office of Research Compliance determined this study not to be human subject research (Protocol 2026–00063). Five MSIG-designed scenarios on open femur fracture, preeclampsia, tricyclic antidepressant intoxication, alcohol withdrawal, and epidural hematoma were selected from the medical school’s simulation center archives. MSIG are specialty-focused extracurricular student organizations that can design and run supplemental simulation-based student events. MSIG scenarios were designed by medical students with no simulation design experience, under unstructured guidance from specialty and simulation faculty, through iterative in-person meetings and email communications over the course of months. All MSIG scenarios followed the simulation center’s standardized, 18-component design template (Appendix 1) and were used in extracurricular MSIG simulations.

Three second-year medical student scenario designers (MSSD) created AIG scenarios on the same MSIG SBMS topics. MSSD had participated as learners in standard curricular SBMS but had no prior experience in scenario design, were not involved in the MSIG scenario design, nor had they participated in any of the MSIG SBMS. MSSD accessed MSIG scenario synopses and learning objectives only, without viewing the full MSIG scenario design templates. AIG-assisted scenario design began with an initial prompt in ChatGPT v3.5 (OpenAI) based on the MSIG scenario synopsis (Appendix 2).⁷ MSSD iteratively refined prompts until sufficient information was generated to complete the scenario design template. No files were uploaded with the prompts, no custom generative pretrained transformers (GPTs) were used, and no faculty assistance was provided during the AIG design process.

Three external physician faculty with simulation expertise rated the 10 scenarios in a unique, randomized sequence. Raters were blinded to the study aims and scenario group. Scenarios were provided in the identical scenario design template for both groups, though scenario length was not formally assessed. Raters used the Simulation Scenario Evaluation Tool (SSET) to assess scenario quality.⁸ The SSET included six elements with a total of 20 items: Learning Objectives, Clinical Context/Scenario Overview, Critical Actions, Patient States, Scenario Materials and Resources, and Debriefing Plan. Each element had two to seven items, scored on a scale of 1 (low) to 5 (high). Raters were provided three practice scenarios, separate from the study scenarios, to familiarize themselves with the SSET. No additional rater training was provided.

We calculated the mean scenario generation time and number of prompts for the AIG scenarios. We used paired t tests and Wilcoxon signed-rank tests to compare SSET scores between groups. We calculated the intraclass correlation coefficient (ICC) for interrater reliability with the intent that the rater’s score would be averaged per scenario. We specified a two-way random effects model because a fixed number of raters score a fixed set of scenarios. Statistical significance was set at P<0.05. All analyses were performed in R version 4.3.0 (R Foundation).

MSIG scenario design required multiple faculty-student meetings over the span of months, along with unmeasured student time. AIG design time was 36.6±18.27 (mean ± SD) minutes, using a mean of 5.8±1.3 prompts per scenario. The individual SSET mean element score in Scenario Materials and Resources was significantly higher in AIG scenarios (3.83±0.20) compared to MSIG scenarios (2.67±0.70; P = 0.017). We found no statistically significant differences between AIG and MSIG SSET overall or other element scores; however, AIG scenarios demonstrated higher mean scores overall and across all elements (Table 1).

The SSET Debriefing Plan element mean scores were lowest for both groups: 2.70±0.61 for AIG and 2.37±1.27 for MSIG. Clinical Content/Scenario Overview demonstrated negative ICC values for agreement (–1.748) and consistency (–1.402), demonstrating a strong disagreement. Within each scenario, the variability between scores was greater than the variability within each rater, leading to large negative ICC values (<–1.0). Debriefing Plan had moderate interrater reliability with an ICC of 0.760 for agreement and 0.759 for consistency. All other elements had low interrater reliability (ICC<0.50).

This pilot study provides support for ChatGPT-assisted novice scenario design. ChatGPT-assisted scenarios surpassed the traditional design process in time-based efficiency. AIG scenarios received higher mean overall scores and in every SSET element; however, only the Scenario Materials and Resources element reached statistical significance. This consistent directionality may represent a signal that the study was underpowered to detect. For example, although the MSSD incorporated MSIG learning objectives into the prompt, the AI also was instructed to generate learning objectives, potentially resulting in clearer or more structured wording, contributing to the differences in Learning Objectives element scores between groups. The findings suggest that novices assisted by AI can generate SBMS with scenario quality comparable to, and possibly higher than, SBMS with student-faculty collaboration. Generative AI may provide a useful starting point for simulation scenario development, especially for novices. However, human SME oversight remains crucial to confirm clinical accuracy and alignment with learning objectives.⁹

Both AIG and MSIG scenarios received their lowest SSET scores in the Debriefing Plan element. AIG debriefing plans consisted of open-ended questions, which may be insufficient to capture the complexity of effective debriefing. These suboptimal ratings suggest that ChatGPT prompts may require enhancement to produce more comprehensive debriefing plans, potentially by incorporating structured frameworks, such as PEARLS or GAS, which are designed to guide debriefers in applying best practices following simulation.^10,11

Student-led SBMS design may serve as an educational activity. Student-led scenario design is underexplored, although a few studies have suggested enhanced learning outcomes for student designers.^12,13 Engaging students in both specialty and primary care scenario design may reinforce traditional learning, fostering a deeper understanding of clinical concepts. Rather than replacing faculty collaboration, AI-assisted design may serve as a starting point from which novice designers or students can engage SME faculty at a more advanced stage, potentially enriching mentorship experiences and clinical reasoning development.

Limitations included a small sample size and a limited number of raters, which reduced statistical power, and the study was not designed or powered to test for equivalence or noninferiority. Findings should be interpreted as preliminary and require replication in larger samples before definitive conclusions can be drawn. Lack of formal rater training and calibration may have contributed to low interrater reliability across SSET elements, including negative ICC values reflecting disagreement among raters. The SSET had limited prior usage and no threshold for defining scenario quality.¹⁴ MSIG scenario design time was not formally measured; the differing methods used to capture design time represent a further limitation. Learning outcomes for MSSD were not evaluated, limiting conclusions about the utility of AI-assisted scenario creation as a learning activity. Future studies should explore the educational impact of AI-assisted scenario design, accounting for AI output variability, and the broader ethical implications of AI use in medical education.

This pilot study supports the potential of simulation novices using AI to streamline the initial SBMS design process. AIG scenarios required less time to create and were comparable in quality to MSIG scenarios, with consistently higher mean scores that did not reach statistical significance. AI offers an efficient tool for making SBMS creation more accessible to students and novice educators, while expanding opportunities for simulation-based learning. The efficiency afforded during initial development may allow novice designers to focus on other critical elements for optimal learning and collaboration with expert simulationists and SMEs.

We thank the simulation experts who participated in this project: Dr. Ismail Saiboon, Dr. Daniel Salcedo, and Dr Len Tanaka.

The project described was supported by the JABSOM Office of the Dean. Kyle M. Ishikawa was partially supported by the U54MD007601 (Ola HAWAII) grant from the National Institute of Health (NIH). The content is solely the responsibility of the authors and does not necessarily represent the official views of NIH.

• Hawaii Professions Education Conference, February 10, 2024, Honolulu, HI
• International Meeting on Simulation in Healthcare, January 13, 2025, Orlando, FL