Do Weekly Alerts From a Mobile Application Influence Reading During Residency?

Each year, more than 1,200 residents sit the family medicine certification examination in Canada. Observational studies have found a relationship between residents’ licensing exam scores and the subsequent quality of their patient care.^1-5 For example, Tamblyn et al reported that family medicine residents achieving higher scores on their certification exam were more likely as family physicians to prescribe fewer contraindicated drugs in practice. Although initial assessment of clinical practice outcomes was limited to the first 18 months of practice, in follow-up study, a sustained relationship was reported between certification exam scores and performance over 4 to 7 years.^3,4 While observational studies generally do not allow for causal inference, residency directors seek proven instructional methods to optimize training and future practice.

As a guide to prepare residents for their certification exam, the College of Family Physicians of Canada (CFPC) developed priority topics and their key features.⁶ In this context, a major challenge for residents is knowledge acquisition and retention.⁷ To address this problem, the strategy of spaced education (SE) emerged from research in the psychology laboratory as a promising solution.⁸ SE takes advantage of both the spacing effect and the testing effect.^9,10 With the spacing effect, clinician-educators can increase retention of information by repeatedly presenting it over time. The testing effect refers to the process of how testing alters the learning process itself to improve knowledge retention, when combined with answer feedback.¹¹

While evidence supports the benefit of SE for residents in highly focused specialties such as urology, we found no published studies of SE in family medicine.^12-16 Outside of urology, we found four small single-site randomized trials of SE in general surgery, oncology, internal medicine and pediatrics.

To our knowledge, residency training in family medicine does not yet leverage the benefits of SE. In line with a push-pull conceptual framework,¹⁷ alerts from a mobile application (app) to exam-relevant clinical information are a type of “nudge” that can raise awareness and stimulate residents to read. Therefore, we predicted that a weekly alert would nudge residents to read about the priority topic of the week. Our objective was to test the feasibility of delivering mobile alerts for one topic per week, to residents in a 2-year residency in family medicine, and to subsequently determine if this intervention had a measurable effect on reading behavior.

App Description

The Information Assessment Method (IAM) mobile app was developed as a tool for residents to prepare for their certification examination. Inspired by the method of spaced education, the app is a content delivery platform with multiple channels and notifications (alerts). Each priority topic contained on average seven key features elaborated by the CFPC. We linked many of these key feature pages to pages of bulleted clinical information, some providing additional links to online knowledge resources. On demand, users could look up any topic through a search function. Screen shots of the app can be viewed at the STFM Resource Library.¹⁸

Participant Description

Participants were recruited in July 2015 from among incoming first year residents at the McGill University and McMaster University family medicine residency programs. Approval for this study, study instruments and consent form was received from the Institutional Review Board of the McGill and McMaster Faculty of Medicine. By email, consenting residents received a username and password to download the app from a web page created for this study. Residents at McMaster University in Ontario were allocated to the intervention group, and received one alert to Priority Topics every Wednesday. Residents at McGill University in Quebec were allocated to the control group and provided with access to an identical version of the app but no weekly alert.

Data Collection

Participants were asked to complete a brief demographic questionnaire. A tracking feature on the app provided data on the following variables: User ID, last name, username, date, and time of visits to key feature pages belonging to each topic, and visits to clinical information pages belonging to each key feature. Key feature page visits within a 7-day time window following an alert to the topic of the week were considered as “alert visits” to that topic.

Data Analysis

After 100 days, we counted the number of page visits to clinical information for each topic, and compared the mean number of page visits per priority topic, at McGill and at McMaster. We performed a paired t-test on these data, using priority topics as pairs.

After 1 year, in the McMaster group we counted the number of times each participant visited the topic of the week, within 7 days of the weekly alert (alert visits). A linear regression model was fit on four data sets (referred to as Models 1, 1b, 2, and 3) using total app visits (page hits) as a dependent variable, and alert visits as a covariate. Model 1 contained all observations. Model 1b consisted of a multiple linear regression that included age, sex, and operating system as covariates. Nine residents did not report this demographic data, and were excluded from this analysis.

Model 2 omitted observations on participants 1, 5, and 7, which were influential outliers as demonstrated by Cook’s Distance. Model 3 omitted observations only from participant 1, as this participant was the only outlier with exceptionally high leverage. The purpose of fitting these models was to ascertain the behavior and sensitivity of these results to the three outliers.

At McMaster, 81 of 96 first-year residents consented and completed a demographic questionnaire. Subsequently, 69 (71%) installed the app. At McGill, 79 of 85 residents consented and completed a demographic questionnaire. Subsequently, 55 (65%) installed the app. In Table 1, we present the participant demographics.

In an exploratory analysis, after the first 100 days, McMaster residents viewed more pages of clinical information across all Priority Topics (1,546 versus 900) and per topic (15.7±21.8 versus 9.1±14.4 pages, P < 0.0003).

After 1 year, 52 weekly alerts were delivered to McMaster residents. These alerts concerned 57 of the 99 Priority Topics, as five of 52 weekly alerts announced two topics in that week. Results of the linear models are displayed in Figure 2.

Each increase of one alert visit was associated with an increase of about three page visits in the app. All four models revealed the "alert visits" variable to be significantly associated with total page visits. For example, Model 1 estimates that on average, an increase of one alert-visit is associated with an increase of 3.2 in the mean of total visits (see Table 2).

In this report, we demonstrate the feasibility of recruiting family medicine residents into a longitudinal study of an app-based educational intervention. Preliminary findings suggest a weekly alert delivered via mobile app is a promising strategy to encourage reading on priority topics. As data collection is ongoing, the effect of a weekly alert awaits the outcome of our final analysis.

These findings are important to inform the design of a definitive trial of spaced education in the family medicine residency. In this definitive trial, we will partner with the CFPC to build a sustainability plan for the IAM app as a non-profit academic initiative. With a potential customer base of more than 1,200 persons entering residency in family medicine each year in Canada, we see a target audience to sustain the app for years to come.

Limitations

This report concerns the first 52 weeks of an 88-week study. Therefore, these results are preliminary. Second, as the IAM app was developed for the expressed purpose of this study, it is not clear to what degree our findings apply to the universe of apps available for exam preparation. Third, during the fifth month of this study, a technical problem prevented data collection for at least 10 days, when an Apple certificate lapsed. This prevented the iOS version from launching.

Next Steps
Phase 1 data collection was complete in April 2017. In phase 2 of this study, we will describe the experience of family medicine residents with respect to receiving a weekly alert to priority topics. This will help us to understand why the intervention did not work well for some residents, while it worked well for others.