Same Content, Different Methods:

Formal didactic lectures are a historic cornerstone of resident education. Core curricular topics are often delivered in the format of a group lecture or an attending-led group discussion. Though these standard methods of delivering information are convenient, new classroom technologies and strategies have been gaining popularity, prompting educators to question which are the most effective and enjoyable strategies to enhance learning. High-fidelity medical simulation training uses advanced technology such as mannequins manipulated by a technician to simulate real-world scenarios such as changes in vital signs. A common use of simulation mannequins is the teaching of procedures, such as endotracheal intubation. Simulation incorporates communication, teamwork, hands-on practice, and standardized assessments of patient care scenarios without risking patient injury or discomfort. Multiple studies comparing high-fidelity simulation training to standard lectures have found simulation improves long-term knowledge retention,1 performance and technical skills,2-8 learner comfort,1,6 and confidence.3 Slide-based lectures, though, have shown similar results to simulation when testing is performed by written assessment.3 Another study showed better immediate recall of material with slide-based lecture when compared to simulation.1 High-fidelity simulations are expensive to implement and perform.

“Engaged classroom” is a term that broadly includes activities that encourage learners to interact with the facilitator during the learning session. The facilitator is no longer simply a performer, as if on stage, but is instead a moderator that may ask learners to answer questions and perform activities that solidify learning during the teaching session. The flipped classroom, in which the student reviews content in total prior to the teaching session, allowing the teaching session to serve only to provide problems or cases to work through to reinforce what the student knows, is one well-publicized form of an engaged classroom. Engaged classrooms improve average test scores, long-term knowledge retention, learner perceptions of lecture effectiveness, and sense of confidence, while decreasing distractibility.9 Studies on asynchronous learning and flipped classrooms show positive results in grades and learner preference compared to a traditional lecture format.10-12 Unlike high-fidelity simulation, engaged classrooms add little in terms of expense compared to slide-based lectures.

Despite multiple studies comparing the traditional slide-based model of lecture to various other techniques, to our knowledge there have been no studies directly comparing the educational value of the three methods: slide-based lecture format, engaged classroom, and high-fidelity simulation. No studies to date have investigated these different techniques in residency education.

Our objective was to identify the most effective method of educational program delivery as measured by knowledge retention at 2 to 4 weeks. This study utilized a collaborative model involving nine residency programs throughout the United States to examine these questions.

Nine family medicine residency programs throughout the United States, all of which had faculty participating in the University of North Carolina Faculty Development Fellowship, collaborated on this educational research project. The family medicine residencies were representative of multiple types of programs, including university-based (two programs), community-based (four programs), and military-based (three programs). The project was reviewed by the Office of Human Research Ethics at University of North Carolina at Chapel Hill (UNC) and deemed IRB-exempt.  

We chose to teach the core topic of dyspnea in a 45-minute session. Teaching materials were created as a standard slide presentation, an engaged classroom activity, and a high-fidelity medical simulation scenario. The standard slide-based presentation utilized PowerPoint, standardized across all three of its teaching sites. For the engaged classroom, the standard slide-based method was modified using Prezi, a web-based interactive presentation platform, for the provision of visual cues and to help guide discussion. The instructor used a case-based, active learning scenario to engage the audience. Learners actively participated in small groups to work through the case, including answering questions or making clinical decisions as the session progressed, with learner decisions and answers impacting the direction of the case. Outcomes of the case varied depending on the learners’ management decisions. For the purposes of our study, the high-fidelity simulation consisted of a mannequin whose clinical response could be manipulated by a technician based on the actions of the learner. The learners evaluated, diagnosed, and treated the mannequin in a simulated hospital environment. This included real-time feedback with standard hospital monitors and equipment. For example, the vital signs would show a hypoxic, tachypneic, tachycardic patient whose vital signs did not improve when treated with oxygen via nasal cannula. The expected action of the learner in this situation was to recommend intubation. Each of the presentations contained the same goals and objectives based on a review of common board questions, a review of the topic of dyspnea,13 and the clinical experience of the investigators.

An assessment tool (test) was created by the investigators, which consisted of 15 questions (Figure 1). In order to assess the test’s discriminant validity, we recruited five experts (board-certified pulmonologists), who each scored 100%. We administered the test among six novices (third-year medical student volunteers), who achieved an average score of 45%. A psychologist reviewed the questions to ensure that inferences could not be made based upon the question stems. Content validity was achieved by ensuring that the test contained items covered during the educational interventions.

Three residency programs were designated as slide-based lecture sites. An instructor provided the teaching based on the slides and included a summary slide with a question and answer session at the end.

Three programs were designated as engaged classrooms with team-based, experiential learning. Learning points were summarized and educational goals were reviewed at the end of the session.

Three residency programs with high-fidelity simulation capability were designated as simulation sites. At the end of the session, learners were debriefed, and the predetermined goals and objectives were reviewed.

Immediately prior to the educational intervention in each group, the learners completed the preintervention assessment (pretest). All instructors in each group utilized an agreed-upon standardized teaching method, and all learners received the same educational article (provided at the times we deemed would be usual for the teaching method utilized). The accompanying article13 to the teaching sessions was delivered at the end of the session for the slide-based learner, during the engaged classroom session for the engaged classroom, and prior to the teaching session with the high-fidelity simulation. During a collaborative work session at the University of North Carolina, the educators from each residency program rehearsed their teaching method to ensure they were standardized and covered the same material.

Teaching sessions at each residency program took place between November 2014 and January 2015, during normally scheduled teaching time. Learners were briefed on the collaborative study and given the opportunity to volunteer as research participants. Informed consent was provided, and the pretest was administered to volunteers. Pretests had unique three-digit codes consisting of the first letter of mother’s maiden name, last digit of cell phone, and last digit of home street address so that it could be matched to posttest results while maintaining anonymity. In addition to the 15 questions, participants were asked to rate their comfort by Likert scale in the management of an acutely dyspneic patient and to provide their postgraduate year of training.

A posttest was administered 2 to 4 weeks after the teaching session, which consisted of the same 15 knowledge assessment items. Learners were again asked to rate their comfort level with the management of dyspnea. Questions were added asking whether the learner had read the article provided, distraction levels, and level of alertness during the teaching time.

Utilizing Stata 13.1 (StataCorp, College Station, TX) for analysis, pairwise comparisons using the t-test were calculated to determine statistical significance. A multivariate analysis was performed to examine baseline differences among groups utilizing SPSS 21 (Armonk, NY: IBM Corp).

A total of 164 residents were recruited to participate in the study; 47 were assigned traditional lecture, 59 engaged classroom, and 58 simulation (Table 1). Learners who consented to and completed the pretest but were late to the session or called out of the session for patient care were not included in the analysis. Multivariate analysis controlling for differences (pretest score, teaching method, pretest comfort level, gender, and postgraduate year [PGY]) showed acceptable levels of multicollinearity. When controlling for all other variables, only pretest scores were predictive of posttest scores. The group assigned to engaged classroom scored lower (62%) on the pretest compared to the groups assigned to slide-based lecture (73%) and simulation (70%).

The primary outcome was change in test scores. A total of 146 learners completed the posttest, which was administered 2 to 4 weeks after the teaching session, for an 89% response rate. There were 47 completers in the slide-based lecture, 53 in the engaged classroom, and 46 in the simulation group. As shown in Figure 2, there was a significant increase in posttest scores in both the engaged classroom group (+11%) and high-fidelity simulation groups (+9%), compared with the slide-based lecture (+6%). The difference between improvements in posttest scores between the simulation and engaged classroom groups was not statistically significant (P=0.2).

Secondary outcomes are shown in Table 2. Learners all reported an increase in comfort in the management of an acutely dyspneic patient, with the largest increase seen in the engaged classroom group. All other secondary outcomes were statistically similar. There were no significant differences in postsession comfort managing patients, perception of educational value, distractibility, or alertness. No one in the slide-based lecture read the article prior to the session; 5% of learners in the engaged classroom reported skimming the article, and in the simulation group, 27% read the article, 44% skimmed it, and 28% did not read it at all.

In this study comparing three different methods of teaching the topic of dyspnea to residents, knowledge assessment scores increased across all three groups. However, learners in the engaged classroom showed a statistically significant improvement compared to traditional slide-based lecture for knowledge retention. Both the engaged classroom and high-fidelity simulation were superior to slide-based lecture in improving learners’ comfort for managing acute dyspnea. Other secondary outcomes did not show statistically significant differences between teaching methods, suggesting that no one method was superior to others in terms of perceived educational value, distractibility, or alertness.

Our study had several strengths. It was a multicenter trial conducted using a cohort of residents across the United States. The programs were diverse and included military and civilian residents. The cohort was also fairly large at 164 residents. Retention of participants was very good with a response rate of 89% for the posttest. The participants were never informed that other sites were using different teaching methods to cover the same topic. Each of the three groups were given the same learning points based on uniform goals and objectives, and the assessment tool was demonstrably learner-appropriate and valid.

A potential limitation of the study is that investigators were unblinded. The assignment of residents to teaching method was not randomized; residents at sites with resources in high-fidelity simulation were defaulted into the high-fidelity simulation group. Additionally, it is difficult to completely control for the effect of specific presenters. It is possible that certain groups performed better because of the interaction with a particular faculty, regardless of the mode of delivery. We attempted to mitigate this effect by using standardized goals and objectives, and also by explicitly omitting any group discussion during the slide portion of the traditional lecture group. It is difficult to determine how the results of this study might carry over into actual patient care. While learners may score higher on the posttest or express an increased level of comfort with one method, it does not necessarily mean that this results in more competent patient care. Finally, we did not include instructor preference of teaching method in our study. This is primarily because the investigators were themselves the instructors.

Notably, the engaged classroom group, while having the highest percentage change in test score, had the lowest pre- and posttest scores. This may have been reflected in the lower reported pretest comfort level in this group. The reason for this difference is uncertain, as all other baseline characteristics were similar among groups. Since this was not a randomized trial, there may be differences between groups that were not measured, which limited our ability to adjust for other potential confounders.

The cost of the teaching methods in this study varies widely. Slide-based lectures and engaged teaching require few resources and can be done with little to no actual cost other than instructor time. Medical simulations, on the other hand, are associated with a potentially high cost. In simulation centers, the cost of the high-fidelity models, the software, and medical supplies can easily exceed $1 million. Additional costs include the personnel who oversee the simulation program and the space the program occupies.

Engaged classroom and high-fidelity simulation often require more preparation and planning than slide-based lectures. Willingness to incorporate alternative teaching methods is unknown. Faculty comfort with and preference of educational format may be important considerations in future studies.

The clinical significance of the differences in teaching methods in our study is unknown. In reality, most teaching is probably a combination of multiple methods. The individual learner’s characteristics, combined with experience, likely affects the long-term retention of material.

Compared to slide-based lecture, both the engaged classroom and the high-fidelity simulation resulted in statistically significant improvement in knowledge retention at 2 to 4 weeks. There were no significant differences in educational value, distractibility, or alertness. Further research may be able to identify the clinical impact of various teaching methods.