Introduction: Prior to the start of the 2020 COVID pandemic, the use of telemedicine among family physicians was limited; telemedicine curriculum in undergraduate and graduate medical education (GME) was even more scarce. In response to the need for training, we developed synchronous and asynchronous versions of a telemedicine curriculum focused on documentation, communication, and virtual physical exam. As the evaluation of the curriculum, this study compares the documentation behaviors of the clinicians participating in the curriculum.
Methods: We compared the documentation practice of asynchronous learners to those participating in synchronous learning over 1 month. We reviewed each clinical note for five practice behaviors: (1) consent for delivery of care via telemedicine, (2) time on the phone, (3) physical examination, (4) procedure code, and (5) billing code.
Results: We reviewed notes from 11 interns (synchronous) and 22 senior residents (asynchronous). Notes written by an intern were significantly more likely to include documentation of consent and a focused exam. Notes written by senior resident were significantly more likely to include documentation of length of the encounter. We detected no significant differences for documenting the billing or procedure code.
Conclusion: Our analysis determined that correct documentation behaviors can be taught through asynchronous mediums. Components requiring effective communication (consent for care and a virtual physical exam) are more effectively taught when there is deliberate practice and immediate feedback on the skills.
Prior to the 2020 COVID pandemic, the use of telemedicine among family physicians was limited; telemedicine curriculum in graduate medical education (GME) was even more scarce.1-8 As practices transitioned to exclusively virtual care, the ACGME rapidly issued support for telemedicine in GME.9 Abruptly, mastering communication in a remote setting, virtual physical exam skills, and appropriate documentation of telemedicine encounters became essential.10-13 In 2019, about 80 medical schools included telemedicine as a curricular topic; there was a lack of data for how many residency programs required telemedicine training.14-16
In our residency program, the overarching need for physician training in telemedicine was noted as a common theme within a process improvement project.17 We created a curriculum focused on practicing communication and physical exam skills in the telephone only setting, and emphasized the correct components for documentation. Due to limitations within our health care system, virtual video visits are not available.
The need for telemedicine-unique training is acknowledged in the literature, and a recent publication of a needs assessment survey highlighted continued gaps in teaching telemedicine.18 Likewise, there is limited evaluation of effective curriculum.19-24 Curricular assessment is vital to ensure learners are achieving the desired objectives. Assessment of the curriculum can occur at four levels as defined by Kirkpatrick: reaction, learning, behavior, and results.25-28 Since the pandemic, curricular assessments, including one objective, structured clinical exam-based assessemnt, have focused on Kirkpatrick level one (reaction) results.29-31
This study describes a rural, academic family medicine residency’s documentation of telemedicine visits and evaluates a just-in-time telemedicine curriculum delivered in two distinct (synchronous/asynchronous) forms in summer 2020. As part of curricular assessment, we examined the difference in behaviors between the initial asynchronous and subsequent, synchronous telemedicine didactics. To our knowledge, this is the first report of how training impacted documentation of a telemedicine visit, a Kirkpatrick level 3 (behavior) evaluation. Our hypothesis was that the behaviors of telemedicine care would be documented more often by learners who participated in synchronous didactics than learners who participated in asynchronous didactics, due to the feedback and deliberate practice components.
Upon the initial transition to telemedicine, a mandatory but unaccountable, asynchronous curriculum was rapidly created and disseminated to senior residents in April 2020. By July 2020, COVID-19 safety measures had been instituted, and telemedicine curriculum was delivered synchronously to incoming interns. The goals and objectives as well as curricular components and educational strategies are outlined in Tables 1 and 2. Strategies including feedback and deliberate practice were utilized to strengthen curricular topics. Table 3 outlines some of the specific considerations for a telephone physical exam.
We compared the documentation practice of asynchronous, self-directed learners to those of synchronous, faculty-directed learners during 1 month (September 8 through October 8, 2020) by randomly selecting three telemedicine visits for each physician. The data abstractors were three individuals who were involved in the creation of the curriculum and/or had directly participated in the curriculum. Each abstractor evaluated a subset of charts from each cohort.
Abstractors reviewed each clinical note for five documentation behaviors taught in the curriculum. Data abstractors coded the clinical notes for presence or absence of five documentation factors: (1) consent for telemedicine (as a marker of communication), (2) time on the phone, (3) physical examination, (4) procedure code for telemedicine care, and (5) correct billing code. These documentation behaviors were evidence of the learner internalizing and implementing the objectives of telemedicine care. To assess physical exam practice, we coded on a three-level variable: complete absence, presence, or presence of complaint-focused exam. Table 4 provides examples.
The Joint Base San Antonio-Lackland Institutional Review Board determined this project to constitute a quality improvement activity, not human subjects research.
The sample included clinical notes from 11 interns (synchronous, faculty-directed version) and 22 senior residents (asynchronous, self-directed version) totaling 99 unique encounters. Two residents were excluded from the data set due to not having completed three virtual visits within this time frame.
In documentation by interns, 73% of encounter notes included consent to complete virtual care compared to 38% for seniors (χ2 [1, N=99]=10.69, P<.001). Encounter notes written by interns included a focused physical exam 58% of the time compared to 13% for senior residents (χ2 [2, N=99]=24.57, P< .001). The results of physical exam documentation are included in Table 4. Interestingly, the reverse was true for documenting length of encounter. Senior residents were significantly more likely to include documentation of length of the virtual encounter (21% compared to 0%, χ2 [1, N=99]=8.15, P<.01). We detected no significant differences for documenting the billing or procedure code. Table 5 provides a summary of results.
In this study of documentation behaviors of telemedicine encounters, differences are notable between the two curricular versions. In the asynchronous version, there was no accountability for completion of the role plays, nor feedback provided for the deliberate practice unlike the synchronous version. This timely feedback was likely instrumental in honing the skill of performing behaviors that required communication with the patient (ie, physical exam and consent for care). Additionally, the accountability of the synchronous version ensured deliberate practice of telemedicine skills.
Components of time documentation and coding guidance were disseminated in other emails outside the curriculum, likely influencing this behavior. The residents who received asynchronous training did so at the same time the coding guidance was emerging, thus an emphasis was made on including the time in the note. Furthermore, these were the senior residents who likely already established coding behaviors as part of their documentation patterns. By the time the interns arrived and received synchronous training, this coding emphasis had waned. For non-communication-based behaviors, emphasis outside of formal training (via email, in huddles, etc) may be necessary to lead to behavior change.
As previously mentioned, this curricular project was implemented within an ongoing process improvement.17 Changes for our practice that came out of this curricular assessment included creation of templates for telemedicine visits (ie, consent, physical exam), annual synchronous training during intern orientation, faculty development, and direct observation of visits.
Of note, telemedicine curricula, including at a national level,32 have been developed in the past 2 years at a rapid pace; this method of sampled chart review provides a mechanism for evaluating other curriculum on a Kirkpatrick 3 (behavior) change.
Limitations of this study include the size of the chart sample, which was collected in one residency program across 1 month. Additionally, the charts reflected encounters of learners with different levels of clinical experience; although none of the learners had been conducting scheduled telemedicine clinic appointments (opposed to a short telephone call in response to a patient inquiry) prior to the pandemic. For this reason, the skill of telemedicine, especially physical examination and documentation, was new to all learners, and the interns (synchronous curriculum) showed more consistent documentation of telemedicine skills. Further limitation of the asynchronous curriculum was lack of accountability for completion. The limitations of a retrospective chart review such as the possibility of incomplete documentation by the physician are noted.
Future work should include faculty physicians in the curriculum delivery and evaluation to ensure they have the knowledge and skills to provide feedback to the learners. Further training and evaluation should include development of communication skills and associated documentation for discussing assessment and plan on the phone. Future research should investigate how telemedicine documentation affects patient outcomes.
- Peabody MR, Dai M, Turner K, Peterson LE, Mainous AG III. Prevalence and factors associated with family physicians providing e-visits. J Am Board Fam Med. 2019;32(6):868-875. doi:10.3122/jabfm.2019.06.190081
- Santos MV, Oliveira DC, Novaes MA. A telehealth strategy for increasing adherence in the treatment of hypertension in primary care. Telemed J E Health. 2013;19(4):241-247. doi:10.1089/tmj.2012.0036
- Russo JE, McCool RR, Davies L. VA Telemedicine: An analysis of cost and time savings. Telemed J E Health. 2016;22(3):209-215. doi:10.1089/tmj.2015.0055
- Moore MA, Coffman M, Jetty A, Klink K, Petterson S, Bazemore A. Family physicians report considerable interest in, but limited use of, telehealth services. J Am Board Fam Med. 2017;30(3):320-330. doi:10.3122/jabfm.2017.03.160201
- Bulik RJ, Shokar GS. Integrating telemedicine instruction into the curriculum: expanding student perspectives of the scope of clinical practice. J Telemed Telecare. 2010;16(7):355-358. doi:10.1258/jtt.2010.090910
- Brown SB, Eberle BJ. Use of the telephone by pediatric house staff: a technique for pediatric care not taught. J Pediatr. 1974;84(1):117-119. doi:10.1016/S0022-3476(74)80570-6
- Lamb MP. Telephone precepting: the development of a curriculum. Teach Learn Med. 2004;16(3):276-278. doi:10.1207/s15328015tlm1603_10
- Caralis P. Teaching residents to communicate: the use of a telephone triage system in an academic ambulatory clinic. Patient Educ Couns. 2010;80(3):351-353. doi:10.1016/j.pec.2010.07.028
- ACGME. ACGME response to COVID-19: clarification regarding telemedicine and ACGME surveys. Accessed May 19, 2020. https://www.acgme.org/newsroom/blog/2020/3/acgme-response-to-covid-19-clarification-regarding-telemedicine-and-acgme-surveys/
- Gomez T, Anaya YB, Shih KJ, Tarn DM. A qualitative study of primary care physicians’ experiences with telemedicine during COVID-19. J Am Board Fam Med. 2021;34(suppl):S61-S70. doi:10.3122/jabfm.2021.S1.200517
- Chang JE, Lindenfeld Z, Albert SL, et al. Telephone vs video visits during COVID-19: safety-net provider perspectives. J Am Board Fam Med. 2021;34(6):1103-1114. doi:10.3122/jabfm.2021.06.210186
- Telemedicine training proves vital during COVID-19 crisis, increasing access to care. Weill Cornell Medicine. April 9, 2020. Accessed September 20, 2020. https://news.weill.cornell.edu/news/2020/04/telemedicine-training-proves-vital-during-covid-19-crisis-increasing-access-to-care
- Mulcare M, Naik N, Greenwald P, et al. Advanced Communication and Examination Skills in Telemedicine: a structured simulation-based course for medical students. MedEdPORTAL. 2020;16:11047. Published 2020 Dec 17. doi:10.15766/mep_2374-8265.11047
- Theobald M, Brazelton T. STFM forms task force to develop a national telemedicine curriculum. Ann Fam Med. 2020;18(3):285-286. doi:10.1370/afm.2549
- Association of American Medical Colleges. Content documentation in required courses and elective courses. Accessed Oct 10, 2019. https://www.aamc.org/data-reports/curriculum-reports/interactivedata/content-documentation-required-courses-and-elective-courses
- Waseh S, Dicker AP. Telemedicine training in undergraduate medical education: mixed-methods review. JMIR Med Educ. 2019;5(1):e12515. doi:10.2196/12515
- Silver SL, Lewis MN, Ledford CJW. A stepwise transition to telemedicine in response to COVID-19. J Am Board Fam Med. 2021;34(suppl):S152-S161. doi:10.3122/jabfm.2021.S1.200358
- Venditti SA, Sazegar P, Fuchs LC, Snarskis CE. Family medicine resident and faculty perceptions about the strengths and limitations of telemedicine training. PRiMER Peer-Rev Rep Med Educ Res. 2022;6:9. doi:10.22454/PRiMER.2022.665996
- Sharma R, Nachum S, Davidson KW, Nochomovitz M. It's not just FaceTime: core competencies for the Medical Virtualist. Int J Emerg Med. 2019 Mar 12;12(1):8. . doi:10.1186/s12245-019-0226-y.
- Phan RCV, Van Le D, Nguyen A, Mader K. Rapid adoption of telehealth at an interprofessional student-run free clinic. PRiMER Peer-Rev Rep Med Educ Res. 2020;4:23. doi:10.22454/PRiMER.2020.241619
- Waseh S, Dicker AP. Telemedicine training in undergraduate medical education: mixed-methods review. JMIR Med Educ. 2019;5(1):e12515. https://mededu.jmir.org/2019/1/e12515/doi:10.2196/12515
- Iancu AM, Kemp MT, Alam HB. Unmuting medical students’ education: utilizing telemedicine during the COVID-19 pandemic and beyond. J Med Internet Res. 2020;22(7):e19667. PMID:32614776 doi:10.2196/19667
- Jumreornvong O, Yang E, Race J, Appel J. Telemedicine and medical education in the age of COVID-19. Acad Med. 2020;95(12):1838-1843. doi:10.1097/ACM.0000000000003711
- Ledford CJW. This is why. Fam Med. 2017;49(4):268-269.
- Kirkpatrick DL. Evaluating Training Programs : the Four Levels. San Francisco : Emeryville, CA :Berrett-Koehler; Publishers Group West; 1994
- Kirkpatrick DL. Techniques for evaluation training programs. J Am Soc Train Dir. 1959;13:21-26.
- Stokking K. Levels of evaluation: Kirkpatrick, Kaufman and Keller, and Beyond. Hum Resour Dev Q. 1996;7(2):179-183. doi:10.1002/hrdq.3920070208
- Praslova L. Adaptation of Kirkpatrick’s four level model of training criteria to assessment of learning outcomes and program evaluation in Higher Education. Educ Assess, Eval Account. 2010;22(3):215-225. doi:10.1007/s11092-010-9098-7
- Savage DJ, Gutierrez O, Montané BE, et al. Implementing a telemedicine curriculum for internal medicine residents during a pandemic: the Cleveland Clinic experience. Postgraduate Medical Journal Published Online First. 2021;10. doi:10.1136/postgradmedj-2020-139228
- Ha E, Zwicky K, Yu G, Schechtman A. Developing a telemedicine curriculum for a family medicine residency. PRiMER Peer-Rev Rep Med Educ Res. 2020;4:21. doi:10.22454/PRiMER.2020.126466
- Martinez L, Holley A, Brown S, Abid A. Addressing the rapidly increasing need for telemedicine education for future physicians. PRiMER Peer-Rev Rep Med Educ Res. 2020;4:16. doi:10.22454/PRiMER.2020.275245
- Telemedicine Online Course. Stfm.org. https://www.stfm.org/teachingresources/curriculum/telemedicine/telemedicinecourses/. Accessed May 4, 2022.
- Academy of Communication in Healthcare. COVID 19 telemedicine: relationship-centered communication skills. Accessed 9 July 2021. https://achonline.org/COVID-19/Telemedicine
- Baker LH, O’connell D, Platt FW. “What else?” Setting the agenda for the clinical interview. Ann Intern Med. 2005;143(10):766-770. doi:10.7326/0003-4819-143-10-200511150-00033