Do Weekly Alerts From a Mobile Application Influence Reading During Residency?

Roland Grad, MD, CM | Pierre Pluye, MD | Eric Wong, MD | Carlos Brailovsky, MD | Jonathan L. Moscovici, MSc | Janusz Kaczorowski, PhD | Charo Rodriguez, MD | Francesca Luconi, PhD | Mathieu Rousseau, MD, CM | Mark Karanofsky, MD, CM | Bethany Delleman, MA | Stefan Kegel, MD | Mathew Mercuri, PhD | Maria Kluchnyk, MD | Inge Schabort, MB, ChB

PRiMER. 2017;1:8.

Published: 8/8/2017 | DOI: 10.22454/PRiMER.2017.243866


Background: The benefits of “spaced education” have been documented for residents in highly focused specialties. We found no published studies of spaced education in family medicine. In this study, we report on the feasibility of delivering weekly alerts from a mobile application (app) developed for exam preparation, to increase the reading of clinical information in the family medicine residency.


Design: This is a 2-phase mixed methods study. Phase one is a quasi-experimental study of resident reading of information related to priority topics in family medicine. Reading was documented by page views in a noncommercial mobile app.

Participants: All incoming first-year residents at two university training programs in Canada. The intervention group received one alert per week to priority topics on the app, beginning in their second month of residency. The control group was given access to the same app, but received no alerts.

Results: In this paper, we report the phase one preliminary findings. In the intervention group, 81 of 96 first year residents consented. At the control site, 79 of 85 residents consented. After 100 days, intervention group residents had viewed more pages of clinical information across all 99 priority topics (1,546 versus 900) and per topic (15.7 versus 9.1 pages, P < 0.0003). On average, each increase of one visit to the app following a weekly alert was associated with an increase of 3.2 visits to pages of clinical information in the app.

Conclusion: A weekly alert delivered via mobile app shows promise with respect to reading in the family medicine residency.


Each year, more than 1,200 residents sit the family medicine certification examination in Canada. Observational studies have found a relationship between residents’ licensing exam scores and the subsequent quality of their patient care.1-5 For example, Tamblyn et al reported that family medicine residents achieving higher scores on their certification exam were more likely as family physicians to prescribe fewer contraindicated drugs in practice. Although initial assessment of clinical practice outcomes was limited to the first 18 months of practice, in follow-up study, a sustained relationship was reported between certification exam scores and performance over 4 to 7 years.3,4 While observational studies generally do not allow for causal inference, residency directors seek proven instructional methods to optimize training and future practice.

As a guide to prepare residents for their certification exam, the College of Family Physicians of Canada (CFPC) developed priority topics and their key features.6 In this context, a major challenge for residents is knowledge acquisition and retention.7 To address this problem, the strategy of spaced education (SE) emerged from research in the psychology laboratory as a promising solution.8 SE takes advantage of both the spacing effect and the testing effect.9,10 With the spacing effect, clinician-educators can increase retention of information by repeatedly presenting it over time. The testing effect refers to the process of how testing alters the learning process itself to improve knowledge retention, when combined with answer feedback.11

While evidence supports the benefit of SE for residents in highly focused specialties such as urology, we found no published studies of SE in family medicine.12-16 Outside of urology, we found four small single-site randomized trials of SE in general surgery, oncology, internal medicine and pediatrics.

To our knowledge, residency training in family medicine does not yet leverage the benefits of SE. In line with a push-pull conceptual framework,17 alerts from a mobile application (app) to exam-relevant clinical information are a type of “nudge” that can raise awareness and stimulate residents to read. Therefore, we predicted that a weekly alert would nudge residents to read about the priority topic of the week. Our objective was to test the feasibility of delivering mobile alerts for one topic per week, to residents in a 2-year residency in family medicine, and to subsequently determine if this intervention had a measurable effect on reading behavior.


App Description

The Information Assessment Method (IAM) mobile app was developed as a tool for residents to prepare for their certification examination. Inspired by the method of spaced education, the app is a content delivery platform with multiple channels and notifications (alerts). Each priority topic contained on average seven key features elaborated by the CFPC. We linked many of these key feature pages to pages of bulleted clinical information, some providing additional links to online knowledge resources. On demand, users could look up any topic through a search function. Screen shots of the app can be viewed at the STFM Resource Library.18

Participant Description

Participants were recruited in July 2015 from among incoming first year residents at the McGill University and McMaster University family medicine residency programs. Approval for this study, study instruments and consent form was received from the Institutional Review Board of the McGill and McMaster Faculty of Medicine. By email, consenting residents received a username and password to download the app from a web page created for this study. Residents at McMaster University in Ontario were allocated to the intervention group, and received one alert to Priority Topics every Wednesday. Residents at McGill University in Quebec were allocated to the control group and provided with access to an identical version of the app but no weekly alert.

Data Collection

Participants were asked to complete a brief demographic questionnaire. A tracking feature on the app provided data on the following variables: User ID, last name, username, date, and time of visits to key feature pages belonging to each topic, and visits to clinical information pages belonging to each key feature. Key feature page visits within a 7-day time window following an alert to the topic of the week were considered as “alert visits” to that topic.

Data Analysis

After 100 days, we counted the number of page visits to clinical information for each topic, and compared the mean number of page visits per priority topic, at McGill and at McMaster. We performed a paired t-test on these data, using priority topics as pairs.

After 1 year, in the McMaster group we counted the number of times each participant visited the topic of the week, within 7 days of the weekly alert (alert visits). A linear regression model was fit on four data sets (referred to as Models 1, 1b, 2, and 3) using total app visits (page hits) as a dependent variable, and alert visits as a covariate. Model 1 contained all observations. Model 1b consisted of a multiple linear regression that included age, sex, and operating system as covariates. Nine residents did not report this demographic data, and were excluded from this analysis.

Model 2 omitted observations on participants 1, 5, and 7, which were influential outliers as demonstrated by Cook’s Distance. Model 3 omitted observations only from participant 1, as this participant was the only outlier with exceptionally high leverage. The purpose of fitting these models was to ascertain the behavior and sensitivity of these results to the three outliers.


At McMaster, 81 of 96 first-year residents consented and completed a demographic questionnaire. Subsequently, 69 (71%) installed the app. At McGill, 79 of 85 residents consented and completed a demographic questionnaire. Subsequently, 55 (65%) installed the app. In Table 1, we present the participant demographics.

In an exploratory analysis, after the first 100 days, McMaster residents viewed more pages of clinical information across all Priority Topics (1,546 versus 900) and per topic (15.7±21.8 versus 9.1±14.4 pages, P < 0.0003).

After 1 year, 52 weekly alerts were delivered to McMaster residents. These alerts concerned 57 of the 99 Priority Topics, as five of 52 weekly alerts announced two topics in that week. Results of the linear models are displayed in Figure 2.

Each increase of one alert visit was associated with an increase of about three page visits in the app. All four models revealed the "alert visits" variable to be significantly associated with total page visits. For example, Model 1 estimates that on average, an increase of one alert-visit is associated with an increase of 3.2 in the mean of total visits (see Table 2).


In this report, we demonstrate the feasibility of recruiting family medicine residents into a longitudinal study of an app-based educational intervention. Preliminary findings suggest a weekly alert delivered via mobile app is a promising strategy to encourage reading on priority topics. As data collection is ongoing, the effect of a weekly alert awaits the outcome of our final analysis.

These findings are important to inform the design of a definitive trial of spaced education in the family medicine residency. In this definitive trial, we will partner with the CFPC to build a sustainability plan for the IAM app as a non-profit academic initiative. With a potential customer base of more than 1,200 persons entering residency in family medicine each year in Canada, we see a target audience to sustain the app for years to come.


This report concerns the first 52 weeks of an 88-week study. Therefore, these results are preliminary. Second, as the IAM app was developed for the expressed purpose of this study, it is not clear to what degree our findings apply to the universe of apps available for exam preparation. Third, during the fifth month of this study, a technical problem prevented data collection for at least 10 days, when an Apple certificate lapsed. This prevented the iOS version from launching.

Next Steps
Phase 1 data collection was complete in April 2017. In phase 2 of this study, we will describe the experience of family medicine residents with respect to receiving a weekly alert to priority topics. This will help us to understand why the intervention did not work well for some residents, while it worked well for others.


  1. Wenghofer E, Klass D, Abrahamowicz M, Dauphinee D, Jacques A, Smee S, et al. Doctor scores on national qualifying examinations predict quality of care in future practice. Medical Education. 2009;43(12):1166-73.
  1. Tamblyn R, Abrahamowicz M, Dauphinee D, Wenghofer E, Jacques A, Klass D, et al. Physician scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities. JAMA. 2007;298(9):993-1001.
  1. Tamblyn R, Abrahamowicz M, Brailovsky C, Grand'Maison P, Lescop J, Norcini J, et al. Association between licensing examination scores and resource use and quality of care in primary care practice. JAMA. 1998;280:989-96.
  1. Tamblyn R, Abrahamowicz M, Dauphinee WD, Hanley JA, Norcini J, Girard N, et al. Association between licensure examination scores and practice in primary care. JAMA. 2002;288:3019-26.
  1. Cadieux G, Abrahamowicz M, Dauphinee D, Tamblyn R. Are physicians with better clinical skills on licensing examinations less likely to prescribe antibiotics for viral respiratory infections in ambulatory care settings? Medical Care. 2011;49(2):156.
  1. Allen T, Brailovsky C, Rainsberry P, Lawrence K, Crichton T, Carpentier M-P, et al. Defining competency-based evaluation objectives in family medicine. Canadian Family Physician. 2011;57:e331-40.

  2. Bell DS, Harless CE, Higa JK, Bjork EL, Bjork RA, Bazargan M, et al. Knowledge retention after an online tutorial: a randomized educational experiment among resident physicians. Journal of General Internal Medicine. 2008;23(8):1164-71.
  1. Carpenter S, Cepeda N, Rohrer N, Kang S, Pashler H. Using spacing to enhance diverse forms of learning: Review of recent research and implications for instruction. Educational Psychology Review. 2012;24(3):369-78.
  1. Cepeda NJ, Vul E, Rohrer D. Spacing effects in learning: a temporal ridgeline of optimal retention. Psychol Sci. 2008;19:1095-102.
  1. Pashler H, Rohrer D, Cepeda NJ, Carpenter SK. Enhancing learning and retarding forgetting: choices and consequences. Psychonomic Bulletin & Review. 2007;14:187-93.
  1. Karpicke JB, JR. Retrieval practice produces more learning than elaborative studying with concept mapping. Science. 2011;331.
  1. Kerfoot BP, DeWolf WC, Masser BA, Church PA, Federman DD. Spaced education improves the retention of clinical knowledge by medical students: a randomised controlled trial. Medical Education. 2007;41(1):23-31.
  1. Kerfoot BP, Baker HE, Koch MO, Connelly D, Joseph DB, Ritchey ML. Randomized, controlled trial of spaced education to urology residents in the United States and Canada. Journal of Urology. 2007;177(4):1481-7.
  1. Kerfoot BP. Learning benefits of on-line spaced education persist for 2 Years. The Journal of Urology. 2009;181(6):2671-3.
  1. Kerfoot BP, Baker H. An Online spaced-education game to teach and assess residents: a multi-institutional prospective trial. Journal of the American College of Surgeons. 2012;214(3):367-73.
  1. Kerfoot BP, Armstrong EG, O'Sullivan PN. Interactive spaced-education to teach the physical examination: A randomized controlled trial. Journal of General Internal Medicine. 2008;23(7):973-8.
  1. Pluye P, Grad RM, Granikov V, Jagosh J, Leung KH. Evaluation of email alerts in practice: Part 1 - review of the literature on clinical emailing channels. Journal of Evaluation in Clinical Practice. 2010;16(6):1227-35.
  1. Grad R. The IAM Mobile App for Spaced Education STFM Resource Library2016 [The IAM mobile app stimulates reflective learning when linked to clinical information derived from "push" (alerting) services or "pull" databases. Available from: Accessed 2017-03-06

Lead Author

Roland Grad, MD, CM

Affiliations: McGill University Department of Family Medicine


Pierre Pluye, MD - McGill University Department of Family Medicine

Eric Wong, MD - The College of Family Physicians of Canada

Carlos Brailovsky, MD - The College of Family Physicians of Canada

Jonathan L. Moscovici, MSc - QuintilesIMS

Janusz Kaczorowski, PhD - Université de Montréal Department of Family and Emergency Medicine

Charo Rodriguez, MD - McGill University Department of Family Medicine

Francesca Luconi, PhD - McGill University Continuing Professional Development Office

Mathieu Rousseau, MD, CM - Mt Sinai Hospital

Mark Karanofsky, MD, CM - McGill University Department of Family Medicine

Bethany Delleman, MA - McMaster University Department of Family Medicine

Stefan Kegel, MD

Mathew Mercuri, PhD - McMaster University Department of Emergency Medicine

Maria Kluchnyk, MD - McGill University Department of Family Medicine

Inge Schabort, MB, ChB - McMaster University Department of Family Medicine

Corresponding Author

Roland Grad, MD, CM

Correspondence: McGill University - Family Medicine 3755 Cote Ste Catherine Road, Montreal, Quebec H3T1E2 Canada. 514-340-8222-5851 Fax: 514-340-8300


Fetching other articles...

Loading the comment form...

Submitting your comment...

There are no comments for this article.

Downloads & Info


Related Content