Text Messaging as a Teaching Tool in a Family Medicine Clerkship Rotation

Preferred learning styles for today’s learners are changing with advances in technology. The use of mobile devices and social media has shifted the way students interact and learn.¹ Text messages as a teaching tool is an emerging strategy that educators may find helpful with distance education and in longitudinal interactions with learners.² Text messaging has provided some undergraduate students positive learning experiences.^3,4 Several studies have evaluated text messages in medical training.^5-9 These messages have been used in many ways including reinforcing teaching points, delivering new content, assessing participant knowledge, providing resources, and focusing attention on interesting subjects. A recent meta-analysis has also shown it can improve self-care in patients with diabetes.¹⁰

In medical education, text messages have been viewed by learners as a good method for data collection in research,¹¹ and weekly text alerts have been shown to increase compliance with reading assignments.¹² Studies on text messaging in medical education have shown mixed results, with some showing improved learning satisfaction, self-perceived learning, or learning based on posttests,^5,6,9 while others have shown no benefit on improved learning comparing pre- and posttests.^7,8 Our study was designed to compare learning with text messages and emails in a focused curriculum on diabetes management.

Our third-year medical students complete a 6-week rural clerkship throughout South Carolina. Before starting on rotation, students participate in a 2-day orientation, which includes a 2-hour diabetes management workshop to review diabetes goals of care, discuss medication management, and practice testing blood glucose.

During the 2015-2016 academic year, students were recruited to a parallel group randomized controlled trial comparing 15 text messages sent throughout rotation to an email with the same content sent during the first week of rotation. Allocation concealment was enforced with sequentially numbered, opaque, sealed envelopes prior to randomization. All participants were encouraged to respond to the messages to support active learning and discussion (Figure 1). To assess knowledge and learner satisfaction, participants completed a 10-question posttest and a six-question survey at the end of rotation (Figure 2).

The primary outcome was a comparison of test scores between the two groups on the 10-question posttest. All questions were evenly weighted with a maximum possible score of 10. Learner satisfaction was a secondary outcome from the six-question survey with Likert scale questions and one open-response question. A secondary safety outcome evaluated scores on two diabetes questions from the fmCASES National Exam. Results were excluded from the analysis if a posttest was not completed or participants failed to receive an intervention.

A power analysis showed at least 12 participants from each group were needed to detect a 25% difference in correctly answered test questions assuming 80% power and an α level of 0.05. The overall number of correct answers were compared with an independent t-test. The answers to each question, correct versus incorrect, were also assessed individually with χ² analyses. All statistics were performed in IBM SPSS, version 24. Themes were grouped and analyzed from survey questions to assess learner satisfaction. The Medical University of South Carolina Institutional Review Board approved this study via an expedited review process.

A total of 119 students were randomized (66 texting and 53 email), but data from 34 participants was excluded from analysis. The per protocol analysis included 34 texting group participants and 51 email group participants. Fewer participants in the texting group were able to participate because of cell phone reception issues (n=13) and failure to sign up to receive messages (n=15). No posttest was completed by four participants in the texting group and two participants in the email group.

The average number of correct answers was 3.32 (SD 1.29) in the texting group and 3.69 (SD 1.53) in the email group. The maximum number of correct answers out of 10 was 7. A higher percentage of participants who answered six or more questions correctly received the email intervention 9.8% (n=5) vs the texting intervention 2.9% (n=1) although this wasn’t statistically significant (P=0.259; Table 1). Individual question analysis showed statistical significance in only one question looking at application of guidelines on statin dosing. While no participants answered this question correctly from the texting group, 11.8% answered it correctly in the email group (χ² [1, N=85]=4.304, P=0.038; Table 2).

Student satisfaction with text messages was higher at 3.68 (SD 0.87) compared to email at 2.02 (SD 0.95) when rating the educational interventions. These results come from the primary survey question that asked participants to rank the intervention they were randomized to on a 1 to 5 Likert scale with 1=poor, 3=average, and 5=excellent (Figure 2). When asked “Do you think text messages offer quality learning opportunities?” 94% in the texting group answered “yes” vs 90% in the email group. No differences were seen on the fmCASES National Exam, with a high proportion of correct answers in each group.

Analysis of open-ended responses from the text messaging group on how text messages could be more effective revealed mixed results, most commonly related to the timing of the text messages and their content. The suggestions for timing varied, but generally recommended sending messages earlier in the day. Participants suggested formatting the text messages either as high-yield facts or requesting questions that they could answer. No students randomized to the email group responded to the email with comments or questions on the content. Approximately 75% of the text messaging group regularly texted back responses.

There was no difference in aggregate test scores for third-year medical students on diabetes management questions between groups. Only one individual question showed statistically significant differences in favor of the email group. This question and several others were likely too difficult because they were set up as complex multiple-choice questions, and resulted in a low number of correct responses. However, on average learners liked interacting with text messages more than an email in this study. A majority of respondents from both groups believed that text messages offered benefits as an educational strategy.

Text messaging provides an opportunity to engage learners remotely as evidenced by the high response rate from the text messaging group. About half of the responses to questions were correct, and the other half presented teachable moments. Even completely correct responses provided opportunities for positive feedback and additional learning tips. The ability to deliver quick, personalized feedback was viewed positively by the investigators.

Several confounding variables may have contributed to our results. The exam was possibly too difficult, with average correct scores of only 3.32 and 3.69, out of 10, in our two groups. Technical challenges with having participants not sign up for text messages or having cell phone reception issues led to many students being unable to engage with our intervention. It was unclear if participants in either group caused cross-contamination of our data by forwarding on messages to other participants. Moreover, while multiple-choice questions may represent a convenient and repeatable measure of learning, they may not reflect true competence in providing quality patient care.¹³ Lastly, email messages were sent out only in the beginning of rotation rather than spaced out over 5 weeks like the text messages because of limitations in prescheduling emails.

A text messaging curriculum likely needs to be adapted to the learner. Many participants receiving text messages wanted them to be sent at different times. Technical challenges occurred with limited cell phone service for some students and many participants not signing up for text messages. There also is a possibility that neither strategy affected student learning because no pretest was given to assess baseline knowledge and a control group receiving no intervention was not included. Results from this pilot study are being used to improve teaching methods and redesign assessment questions.