Introduction: Today’s learners use multiple forms of social communication, such as text messaging, that offer a promising teaching tool for medical education. The purpose of this study was to evaluate a diabetes care curriculum delivered through text messages for third-year medical students on a rural family medicine clerkship.
Methods: A pilot study of 119 participants were compared in a parallel group randomized controlled trial evaluating medical student learning and satisfaction with text messages throughout rotation compared to an email with the same content in their first week of rotation. Participants completed a 10-question multiple-choice test and six survey questions upon completing the rotation. The primary outcome was a difference between test scores among the two groups, and student satisfaction with the educational intervention was a secondary outcome.
Results: A total of 85 participants successfully completed the study protocol (34 text messages and 51 email) and were included in a per protocol analysis. The average number of correct responses per test was 3.32 (SD 1.29) in the texting group and 3.69 (SD 1.53) in the email group (P=0.259). Student satisfaction with text messages was 3.68 (SD 0.87) compared to email at 2.02 (SD 0.95) when rating the educational intervention on a 1 to 5 Likert scale (1=poor, 3=average, and 5=excellent).
Conclusions: Participant knowledge on a challenging posttest was not improved with text messages compared to an email in this pilot study. Satisfaction with text messages was primarily positive. Further study is needed to determine the effectiveness of this content delivery method.
Preferred learning styles for today’s learners are changing with advances in technology. The use of mobile devices and social media has shifted the way students interact and learn.1 Text messages as a teaching tool is an emerging strategy that educators may find helpful with distance education and in longitudinal interactions with learners.2 Text messaging has provided some undergraduate students positive learning experiences.3,4 Several studies have evaluated text messages in medical training.5-9 These messages have been used in many ways including reinforcing teaching points, delivering new content, assessing participant knowledge, providing resources, and focusing attention on interesting subjects. A recent meta-analysis has also shown it can improve self-care in patients with diabetes.10
In medical education, text messages have been viewed by learners as a good method for data collection in research,11 and weekly text alerts have been shown to increase compliance with reading assignments.12 Studies on text messaging in medical education have shown mixed results, with some showing improved learning satisfaction, self-perceived learning, or learning based on posttests,5,6,9 while others have shown no benefit on improved learning comparing pre- and posttests.7,8 Our study was designed to compare learning with text messages and emails in a focused curriculum on diabetes management.
Our third-year medical students complete a 6-week rural clerkship throughout South Carolina. Before starting on rotation, students participate in a 2-day orientation, which includes a 2-hour diabetes management workshop to review diabetes goals of care, discuss medication management, and practice testing blood glucose.
During the 2015-2016 academic year, students were recruited to a parallel group randomized controlled trial comparing 15 text messages sent throughout rotation to an email with the same content sent during the first week of rotation. Allocation concealment was enforced with sequentially numbered, opaque, sealed envelopes prior to randomization. All participants were encouraged to respond to the messages to support active learning and discussion (Figure 1). To assess knowledge and learner satisfaction, participants completed a 10-question posttest and a six-question survey at the end of rotation (Figure 2).
The primary outcome was a comparison of test scores between the two groups on the 10-question posttest. All questions were evenly weighted with a maximum possible score of 10. Learner satisfaction was a secondary outcome from the six-question survey with Likert scale questions and one open-response question. A secondary safety outcome evaluated scores on two diabetes questions from the fmCASES National Exam. Results were excluded from the analysis if a posttest was not completed or participants failed to receive an intervention.
A power analysis showed at least 12 participants from each group were needed to detect a 25% difference in correctly answered test questions assuming 80% power and an α level of 0.05. The overall number of correct answers were compared with an independent t-test. The answers to each question, correct versus incorrect, were also assessed individually with χ2 analyses. All statistics were performed in IBM SPSS, version 24. Themes were grouped and analyzed from survey questions to assess learner satisfaction. The Medical University of South Carolina Institutional Review Board approved this study via an expedited review process.
A total of 119 students were randomized (66 texting and 53 email), but data from 34 participants was excluded from analysis. The per protocol analysis included 34 texting group participants and 51 email group participants. Fewer participants in the texting group were able to participate because of cell phone reception issues (n=13) and failure to sign up to receive messages (n=15). No posttest was completed by four participants in the texting group and two participants in the email group.
The average number of correct answers was 3.32 (SD 1.29) in the texting group and 3.69 (SD 1.53) in the email group. The maximum number of correct answers out of 10 was 7. A higher percentage of participants who answered six or more questions correctly received the email intervention 9.8% (n=5) vs the texting intervention 2.9% (n=1) although this wasn’t statistically significant (P=0.259; Table 1). Individual question analysis showed statistical significance in only one question looking at application of guidelines on statin dosing. While no participants answered this question correctly from the texting group, 11.8% answered it correctly in the email group (χ2 [1, N=85]=4.304, P=0.038; Table 2).
Student satisfaction with text messages was higher at 3.68 (SD 0.87) compared to email at 2.02 (SD 0.95) when rating the educational interventions. These results come from the primary survey question that asked participants to rank the intervention they were randomized to on a 1 to 5 Likert scale with 1=poor, 3=average, and 5=excellent (Figure 2). When asked “Do you think text messages offer quality learning opportunities?” 94% in the texting group answered “yes” vs 90% in the email group. No differences were seen on the fmCASES National Exam, with a high proportion of correct answers in each group.
Analysis of open-ended responses from the text messaging group on how text messages could be more effective revealed mixed results, most commonly related to the timing of the text messages and their content. The suggestions for timing varied, but generally recommended sending messages earlier in the day. Participants suggested formatting the text messages either as high-yield facts or requesting questions that they could answer. No students randomized to the email group responded to the email with comments or questions on the content. Approximately 75% of the text messaging group regularly texted back responses.
There was no difference in aggregate test scores for third-year medical students on diabetes management questions between groups. Only one individual question showed statistically significant differences in favor of the email group. This question and several others were likely too difficult because they were set up as complex multiple-choice questions, and resulted in a low number of correct responses. However, on average learners liked interacting with text messages more than an email in this study. A majority of respondents from both groups believed that text messages offered benefits as an educational strategy.
Text messaging provides an opportunity to engage learners remotely as evidenced by the high response rate from the text messaging group. About half of the responses to questions were correct, and the other half presented teachable moments. Even completely correct responses provided opportunities for positive feedback and additional learning tips. The ability to deliver quick, personalized feedback was viewed positively by the investigators.
Several confounding variables may have contributed to our results. The exam was possibly too difficult, with average correct scores of only 3.32 and 3.69, out of 10, in our two groups. Technical challenges with having participants not sign up for text messages or having cell phone reception issues led to many students being unable to engage with our intervention. It was unclear if participants in either group caused cross-contamination of our data by forwarding on messages to other participants. Moreover, while multiple-choice questions may represent a convenient and repeatable measure of learning, they may not reflect true competence in providing quality patient care.13 Lastly, email messages were sent out only in the beginning of rotation rather than spaced out over 5 weeks like the text messages because of limitations in prescheduling emails.
A text messaging curriculum likely needs to be adapted to the learner. Many participants receiving text messages wanted them to be sent at different times. Technical challenges occurred with limited cell phone service for some students and many participants not signing up for text messages. There also is a possibility that neither strategy affected student learning because no pretest was given to assess baseline knowledge and a control group receiving no intervention was not included. Results from this pilot study are being used to improve teaching methods and redesign assessment questions.
The researchers thank Marti Sturdevant, Sarah McIntyre, Jennifer Gavin, Jessica Bright, Justin Wilson, Sarah Fisher, and Carole Berini for their significant efforts with research coordination, statistical analyses, and data collection with this study.
Financial Support: The South Carolina Clinical and Translational Research (SCTR) Institute provided voucher funds of $1,000 to help fund participant remuneration for this pilot study.
Portions of this article content were previously presented:
- Bragg S, Watson KH, Chessman A. Teaching to Text: Preliminary Results and Tutorial. 2016 Society of Teachers of Family Medicine Annual Spring Conference. April 30-May 4, 2016, Minneapolis, MN.
- Bragg S, Watson KH. Getting the Message Across: Teaching Diabetes Management Through Text Messages. 2015 STFM Annual Spring Conference. April 25-29, 2015. Orlando, FL.
- Rossing JP, Miller WM, Cecil AK, Stamper SE. iLearning: the future of higher education? Student perceptions on learning with mobile tablets. J Scholarsh Teach Learn. 2012;12(2):1-26.
- Broom MA, Adamson GT, Draper LR. Text messaging in medical education. Pediatrics. 2014;133(3):e491-e493. https://doi.org/10.1542/peds.2013-1529
- Jones G, Edwards G, Reid A. How can mobile SMS communication support and enhance a first year undergraduate learning environment? Research in Learning Technology. 2009;17(3):201-218. https://doi.org/10.1080/09687760903247625
- Markett C, Arnedillo Sánchez I, Weber S, Tangney B. Using short message service to encourage interactivity in the classroom. Comput Educ. 2006;46(3):280-293. https://doi.org/10.1016/j.compedu.2005.11.014
- Richardson A, Littrell OM, Challman S, Stein P. Using text messaging in an undergraduate nursing course. J Nurs Educ. 2011;50(2):99-104. https://doi.org/10.3928/01484834-20101230-04
- Alipour S, Moini A, Jafari-Adli S, Gharaie N, Mansouri K. Comparison of teaching about breast cancer via mobile or traditional learning methods in gynecology residents. Asian Pac J Cancer Prev. 2012;13(9):4593-4595. https://doi.org/10.7314/APJCP.2012.13.9.4593
- Mount HR, Zakrajsek T, Huffman M, et al. Text messaging to improve resident knowledge: a randomized controlled trial. Fam Med. 2015;47(1):37-42.
- Hoonpongsimanont W, Kulkarni M, Tomas-Domingo P, et al. Text messaging versus email for emergency medicine residents’ knowledge retention: a pilot comparison in the United States. J Educ Eval Health Prof. 2016;13:36. https://doi.org/10.3352/jeehp.2016.13.36
- Gill CJ, Le NB, Halim N, et al. mCME project V.2.0: randomised controlled trial of a revised SMS-based continuing medical education intervention among HIV clinicians in Vietnam. BMJ Glob Health. 2018;3(1):e000632.
- Saffari M, Ghanizadeh G, Koenig HG. Health education via mobile text messaging for glycemic control in adults with type 2 diabetes: a systematic review and meta-analysis. Prim Care Diabetes. 2014;8(4):275-285. https://doi.org/10.1016/j.pcd.2014.03.004
- Wendling AL, Wudyka AE, Phillips JP, et al. RU4PC? Texting to quantify feedback about primary care and its relationship with student career interest. Fam Med. 2016;48(1):21-29.
- Grad R, Pluye P, Wong E, et al. Do weekly alerts from a mobile application influence reading during residency? PRiMER Peer-Rev Rep Med Educ Res. 2017;1:8. https://doi.org/10.22454/PRiMER.2017.243866
- Epstein RM, Hundert EM. Defining and assessing professional competence. JAMA. 2002;287(2):226-235. https://doi.org/10.1001/jama.287.2.226