Introduction: We developed a new channel on a mobile app as a continuing education tool to augment the use of deprescribing guideline content in clinical practice. In this research brief, we describe the reach and adoption of channel content, as well as user feedback.
Methods: Using Google Analytics, we counted page views of the website (deprescribing.org) where the app was promoted. We calculated total app downloads, monthly active users, and guideline-specific page views. Users were invited to complete the embedded Information Assessment Method (IAM) Questionnaire to obtain feedback on the value of information presented on the Deprescribing Channel.
Results: Between March 2, 2019 and November 30, 2019, we documented 9,454 page views of the promotional web page across 40 countries. The Deprescribing Channel was downloaded 3,256 times with an average of 464 monthly users. In total, the guidelines on this channel were accessed 14,377 times with 49,721 views across all guideline pages. Thirty-seven IAM questionnaires were completed. Thirty-two responses indicated this deprescribing information was relevant for at least one of their patients. Regarding educational outcomes, 22 responses were of learning something new and/or being motivated to learn more.
Conclusion: We documented international interest in a mobile app providing continuing education on deprescribing. App users generated sustained page views over the study period. Feedback from a small number of users was positive with the majority finding the content relevant, educational, and applicable to patient care. Further work is needed to improve the usability of the embedded feedback questionnaire and to evaluate its value in supporting learning.
Polypharmacy and inappropriate medication use is prevalent in older adults and is associated with significant morbidity.1-3 Withdrawal and/or dose reduction (deprescribing) of medications that are no longer needed, or may be causing more harm than benefit, is an essential part of optimizing medication use and is necessary to reduce medication-related harm.4,5 However, clinicians have reported significant barriers to deprescribing in regular practice; a lack of resources (such as guidelines) to identify medications that are suitable for deprescribing, and provide guidance on how to stop them has been highlighted as an important barrier.6,7 Evidence-based deprescribing guidelines support clinicians in conducting shared decision-making about when and how to reduce or stop medications; members of our team collaborated in the development of five such guidelines.8-12 Accompanying two-page open-access decision-support PDF algorithms were developed to support implementation of the guidelines13 (www.deprescribing.org). These algorithms contain only highlights of the guideline presented in a static, noninteractive format.
Clinical decision support mobile apps can instantly connect clinicians to a wealth of information, with the advantages of being portable, easy to use, customizable, low cost and accessible at the point of care on a device that is ubiquitous.14,15 These apps have the potential to improve decision making and clinical outcomes.14-16 In response to requests from users of our guidelines, we developed a Deprescribing Channel on the Information Assessment Method (IAM) Medical Guidelines app to augment the use of the deprescribing guidelines, maximize point-of-care uptake by clinicians, and provide the ability to connect to the more detailed published guideline content and supporting resources as continuing education. The aim of this study was to evaluate the value of this Deprescribing Channel. Here, we present data reflecting the reach and adoption of the channel, as well as user feedback on the value of this information.
In 2015, Roland Grad, MD, developed the IAM Medical Guidelines app to enhance access to a curated selection of recommendations from guidelines relevant to clinicians in primary health care. This free app consists of multiple channels. Each channel is designed specifically for one clinical practice guideline. Our team chose this existing platform for the deprescribing guidelines because of reputation, low cost, and an ongoing collaboration with the developers. Together, we created a Deprescribing Channel containing five algorithms and information on the five drug class-specific deprescribing guidelines.
Beginning with the proton pump inhibitor guideline, a content expert worked with a programmer to develop a logic flow. This consisted of selecting content and response options for personalization across pages, including pop-up boxes with concise deprescribing advice and hyperlinks to full guideline content (Figure 1).
Next, we sought feedback on content, functionality, and user experience from stakeholders including guideline development team members, three individuals with expertise in app design, and 15 attendees at two health care conferences. The next four guideline topics (benzodiazepine receptor agonists, antipsychotics, antihyperglycemics, cholinesterase inhibitors/memantine) were designed with a similar logic flow to ensure a consistent navigation experience. Content review proceeded with members from each relevant guideline development team and representatives from the Canadian Agency for Drugs and Technologies in Health.
The Deprescribing Channel was launched in March 2019 on the App Store and Google Play stores. Users downloaded the IAM Medical Guidelines app to their smartphones and then created an account, after which they could download the Deprescribing Channel and access any of the five algorithms and related deprescribing guidelines information.
A multifaceted promotion strategy included highlighting the app on a dedicated webpage at deprescribing.org, an email banner in research staff signature blocks, social media announcements through the deprescribing.org Facebook and Twitter accounts to over 9,500 followers, quarterly newsletters and e-blasts to subscribers and targeted emails to stakeholder organizations inviting them to share details with their networks, and showcasing the Deprescribing Channel at conferences and events through poster presentations and exhibitor booths.
We conducted data collection from March 2, 2019 to November 30, 2019. Using Google Analytics and page view metrics, we tracked app reach by the number of people accessing the web page and their residing country.
We counted the total number of downloads of the Deprescribing Channel as well as active users (users who accessed the channel at least once during each calendar month). We also examined page views for each guideline algorithm to determine how frequently guideline content was accessed.
The IAM Questionnaire (IAM-v 2014 for clinicians17) added a theory-based and validated measure to assess the value of this deprescribing information. This questionnaire was embedded into the app during the design phase and presented to the user via a link called “Feedback about this algorithm.” This feedback link appeared each time the user tapped the “Finish” button. Demographic questions (including professional designation) and a question asking why the information was sought were added. Other than completion of viewing content within a guideline algorithm, there were no other restrictions to accessing the IAM questionnaire. Users could complete the questionnaire any time they used the app in order to capture their experience in each unique context (eg, different patient, different moment in time, or different guideline leading to a new experience with this information). Additional information and versions of the IAM questionnaire can be found in the STFM Resource Library.17
We indirectly assessed outcomes at the four levels of the Kirkpatrick model,18 with levels 2-4 measured through the IAM questionnaire: Level 1: Reaction—at this level, we measured app downloads and page views; Level 2: Learning—we measured Level 2 through the IAM question on “cognitive impact.” This subconstruct of the IAM questionnaire includes an item on learning. Level 3: Behavior—we measured through the IAM question on the “use or application of information in clinical practice.” Level 4: Results—we measured through the IAM question on “expected benefits for patients,” conditional on this information being used in clinical practice.
The Bruyère Continuing Care Research Ethics Board reviewed and approved this research to use implied consent. Whenever a user completed an IAM questionnaire, they were shown a statement identifying the IAM questionniare as an evaluation tool.
Most views of the web page promoting the app occurred in North America (3,499 in the United States, 2,569 in Canada), with international interest demonstrated by page views across six continents and 40 countries in total (Table 1).
The Deprescribing Channel was downloaded 3,256 times (based on unique user ID numbers). We documented an average of 464 monthly active users. Of the active users, 78% were new downloads and 22% were returning visitors (those who had initiated a prior session within the Channel).
The proton pump inhibitor topic was viewed most frequently, while the cholinesterase inhibitors/memantine topic was viewed the least (Table 2).
A mixed group of 34 app users completed 37 questionnaires; the group breakdown was physicians (15), pharmacists (13), nurses (2) and others (7). Three of those users completed the IAM questionnaire on two occasions. The top three reasons for seeking deprescribing information were to (1) exchange information with colleagues (18), (2) satisfy curiosity (17), and (3) fulfill a personal educational objective (16; Figure 2). In 32 of the 37 IAM questionnaires, respondents reported that the information was partially or totally relevant for at least one of their patients. The majority said that they learned something new (n=22), were motivated to learn more (n=22), or confirmed their actions were correct (n=19) as a result of the information (Figure 3). In 29 questionnaires, respondents said they would use or possibly use this information for a patient, with 18 reporting that they expected a health benefit by applying this information.
We developed an interactive app containing information from five deprescribing guidelines and achieved international dissemination with almost 500 active monthly users, over a 9-month period. Feedback from a small number of users suggests the app content is relevant, educational, and applicable to patients. To our knowledge, this is the first evaluation of a mobile app providing guideline-based information on deprescribing.
While the number of app downloads was high, the number of downloads was less (approximately one-third) than the web page views. The website is designed to have broad appeal and it seems unlikely that all viewers would download a deprescribing guideline app designed for health care professionals. For example, nonclinical members of the general public have full access to the site, but it is unlikely those who are not health professionals would download the application. We are unable to calculate an exact conversion rate (defined as the ratio of website visitors who complete a desired action, in this case downloading the app). However, conversion rates in the e-commerce environment are typically low, with the top 10% of companies typically achieving an 11% conversion rate, at most.19
The IAM questionnaire provided the opportunity to systematically document users’ reflections on deprescribing information in the app. In other work, the IAM questionnaire has enhanced reflective learning, evaluation of knowledge resources, and two-way knowledge exchange between information users and information providers.20 Our results suggest the app provides medical learners with new information for their continuing education and motivates them to learn more about deprescribing. However, only a small proportion of those who downloaded the Deprescribing Channel completed the IAM questionnaire. Two reasons may explain this observation. First, the questionnaire link was restricted to users who clicked on the “Finish” button and it may not have been obvious that this linked to an evaluation questionnaire. Secondly, we provided no incentive for questionnaire completion, such as continuing medical education credits or a contest to win a prize. With such a small sample size, and due to the self-reported nature of the questionnaire, our data may be biased if it was received from those who were more engaged, or had more favorable opinions. In the future, a usability study of this app should consider how and when to better present surveys to maximize user feedback; for example, the questionnaire link button could be made more appealing to click (using wording like “Help us improve the app”). The process also did not allow a measurement of learning through pre- or posttests of knowledge.
Promising avenues for future research include identifying minimal numbers of completed questionnaires needed to calculate a relevance index using the IAM questionnaire,21 approaches to maximize response, measuring app retention through continued visits, and exploring how the use of the Deprescribing Channel in clinical practice compares to the traditional static presentation of text in a guideline. Additionally, a recently published instrument designed to be used by health professional educators could assist in rating the value of the app to support learning in four thematic areas: teaching and learning, user-centered, professional measures and usability.22
The Deprescribing Channel achieved international reach and adoption with app users making sustained page views over a 9-month study period. Feedback from a small number of users was positive, with most finding the content to be educational and applicable to patient care. Despite the low number of completed questionnaires, the value of information in this app can be inferred from a sustained number of monthly users.
The authors acknowledge the following people involved in proposal writing, and app review: Rachel Grant, Jennifer Boswell, Zach Dumont, Kelly Grindrod, Jennifer Hartell, Hannah Irving, Heera Sen, and Steve Smith. They also acknowledge the assistance of Julian Tuck of Refacta.ca in the design and development of the Deprescribing Channel, and the assistance of Leah Clement in manuscript formatting and submission.
Financial Support: This work was supported by a grant from the Centre for Aging + Brain Health Innovation (CABHI): SPARK-3-00173.
Dr Emily Reeve is supported by an Australian National Health and Medical Research Council (NMHRC) – Australian Research Council (ARC) Dementia Research Development Fellowship (APP1105777).
Presentations: This research was presented as a Research in Progress poster at the North American Primary Care Research Group annual meeting in Toronto, Ontario, Canada, November 2019.
Conflicts of Interest: Pam Howell and Tammie Quast received salary support from the grant. The other authors report no conflicts of interest. The IAM Medical Guidelines app is a nonprofit initiative and content is freely available to users.
- Steinman MA. Polypharmacy - time to get beyond numbers. JAMA Intern Med. 2016;176(4):482-483. doi:10.1001/jamainternmed.2015.8597
- Fried TR, O’Leary J, Towle V, Goldstein MK, Trentalange M, Martin DK. Health outcomes associated with polypharmacy in community-dwelling older adults: a systematic review. J Am Geriatr Soc. 2014;62(12):2261-2272. doi:10.1111/jgs.13153
- Passarelli MCG, Jacob-Filho W, Figueras A. Adverse drug reactions in an elderly hospitalised population: inappropriate prescription is a leading cause. Drugs Aging. 2005;22(9):767-777. doi:10.2165/00002512-200522090-00005
- Reeve E, Thompson W, Farrell B. Deprescribing: A narrative review of the evidence and practical recommendations for recognizing opportunities and taking action. Eur J Intern Med. 2017;38:3-11. doi:10.1016/j.ejim.2016.12.021
- Garfinkel D, Ilhan B, Bahat G. Routine deprescribing of chronic medications to combat polypharmacy. Ther Adv Drug Saf. 2015;6(6):212-233. doi:10.1177/2042098615613984
- Anderson K, Stowasser D, Freeman C, Scott I. Prescriber barriers and enablers to minimising potentially inappropriate medications in adults: a systematic review and thematic synthesis. BMJ Open. 2014;4(12):e006544. doi:10.1136/bmjopen-2014-006544
- Ailabouni NJ, Nishtala PS, Mangin D, Tordoff JM. Challenges and enablers of deprescribing: a general practitioner perspective. PLoS One. 2016;11(4):e0151066. doi:10.1371/journal.pone.0151066
- Farrell B, Pottie K, Thompson W, et al. Deprescribing proton pump inhibitors: evidence-based clinical practice guideline. Can Fam Physician. 2017;63(5):354-364.
- Farrell B, Black C, Thompson W, et al. Deprescribing antihyperglycemic agents in older persons: evidence-based clinical practice guideline. Can Fam Physician. 2017;63(11):832-843.
- Pottie K, Thompson W, Davies S, et al. Deprescribing benzodiazepine receptor agonists: evidence-based clinical practice guideline. Can Fam Physician. 2018;64(5):339-351.
- Bjerre LM, Farrell B, Hogel M, et al. Deprescribing antipsychotics for behavioural and psychological symptoms of dementia and insomnia: evidence-based clinical practice guideline. Can Fam Physician. 2018;64(1):17-27.
- Reeve E, Farrell B, Thompson W, et al. Evidence-based Clinical Practice Guideline for Deprescribing Cholinesterase Inhibitors and Memantine. http://sydney.edu.au/medicine/cdpc/resources/deprescribing-guidelines.php. Published 2018. Accessed March 8, 2020.
- Gagliardi AR, Brouwers MC. Do guidelines offer implementation advice to target users? A systematic review of guideline applicability. BMJ Open. 2015;5(2):e007047. doi:10.1136/bmjopen-2014-007047
- Martínez-Pérez B, de la Torre-Díez I, López-Coronado M, Sainz-de-Abajo B, Robles M, García-Gómez JM. Mobile clinical decision support systems and applications: a literature and commercial review. J Med Syst. 2014;38(1):4. doi:10.1007/s10916-013-0004-y
- Watson HA, Tribe RM, Shennan AH. The role of medical smartphone apps in clinical decision-support: A literature review. Artif Intell Med. 2019;100:101707. doi:10.1016/j.artmed.2019.101707
- Ramos D, Grad R, Saroyan A, Nugus P. Seeking coherence between ‘mobile learning’ applications and the everyday lives of medical residents. Perspect Med Educ. 2019;8(3):152-159. doi:10.1007/s40037-019-0519-0
- Information Technology Primary Care Research Group. The Information Assessment Method. Department of Family Medicine, McGill University. 2020. STFM Resource Library. https://resourcelibrary.stfm.org/resourcelibrary/viewdocument/information-impact-assessment-metho?CommunityKey=2751b51d-483f-45e2-81de-4faced0a290a&tab=librarydocuments#ItemCommentPanel. Accessed September 24, 2020.
- The Kirkpatrick Partners. The Kirkpatrick Model. http://www.kirkpatrickpartners.com/Our-Philosophy/The-Kirkpatrick-Model. Accessed August 31, 2020.
- Kim L. What’s a good conversion rate? (It’s higher than you think) The Wordstream Blog https://www.wordstream.com/blog/ws/2014/03/17/what-is-a-good-conversion-rate. Published February 26, 2020. Accessed July 28, 2020.
- Carroll JC, Grad R, Allanson JE, et al. The gene messenger impact project: an innovative genetics continuing education strategy for primary care providers. J Contin Educ Health Prof. 2016;36(3):178-185. doi:10.1097/CEH.0000000000000079
- Galvao MCB, Ricarte ILM, Grad RM, Pluye P. The Clinical Relevance of Information Index (CRII): assessing the relevance of health information to the clinical practice. Health Info Libr J. 2013;30(2):110-120. doi:10.1111/hir.12021
- Gladman T, Tylee G, Gallagher S, Mair J, Rennie SC, Grainger R. A Tool for Rating the Value of Health Education Mobile Apps to Enhance Student Learning (MARuL): Development and Usability Study. JMIR Mhealth Uhealth. 2020;8(7):e18015. doi:10.2196/18015