Because Life Is Open Book: An Open Internet Family Medicine Clerkship Exam

In today’s clinical practice, the ability to find the right answer to a point-of-care question is more important than memorizing facts destined to change. A curriculum in information mastery,¹ the application of evidence-based medicine in clinical practice, can improve residents’ abilities in this area² but current literature lacks formal study of medical students’ information mastery competency.

Many medical schools measure knowledge after the family medicine clerkship via the National Board of Medical Examiners (NBME) subject exam. This exam, a multiple-choice, knowledge-based assessment rather than a competency-based assessment, is not designed to evaluate ability to render real-time clinical judgments using web resources. Open resource exam proponents argue that alternative tests can be particularly valuable in assessing higher levels of learning, eg, application rather than rote memorization.³  

Current evidence on open resource exams is mixed. In one study, medical students who took an open book exam had slightly higher test scores, deeper understanding, and reduced anxiety compared to closed book examinees.⁴ Similarly, college students preferred open resource exams, reported less anxiety, and performed slightly better⁵ or no worse⁶ than closed book examinees. A systematic review of 37 studies (mostly on college students) conversely found better performance, but also more preparation time, with closed book exams.³ Although some students prefer open book tests, their study habits can be disorganized⁷ and performance may vary.⁸ All eight systematically reviewed studies that pertained specifically to medical students used knowledge-based multiple-choice question exams, not case-based questions requiring synthesis of material retrieved from medical systematic reviews.³

Hence current research on open resource exams discusses new ways to assess students in an era of quickly changing information, yet it does not specifically address competency assessments of information mastery. The present study adds to the literature by describing the development, implementation and evaluation of a new open-internet medical information mastery competency assessment tool.

At Tufts University School of Medicine, third-year family medicine clerkship students took a computer-based final exam comprising two parts: a closed book knowledge assessment (KA), and a timed open internet information mastery assessment (IMA). The latter requires students to efficiently access online information in order to answer clinical questions (exam details in Table 1), testing higher-order cognitive skills⁹. For each case scenario, examinees provide a clinical recommendation, cite a source of evidence, and identify a strength of recommendation rating¹⁰ (sample question in Figure 1).   

A pilot phase tested feasibility. An early draft of the open internet IMA was administered to two students, one who had taken Family Medicine (whose curriculum includes an information mastery workshop) and one who had not. Based on their feedback, a revised IMA version was administered (no credit) to two separate groups of eight students apiece who had previously passed the family medicine clerkship. Pilot examinees gave feedback on format, time, clarity and question sequencing. Their cognitive interview transcripts were subjected to standard qualitative analysis methodology to extract themes, which informed modifications to the IMA (Table 2).

Both parts of the updated exam version (KA and IMA) were then administered for high-stakes credit to 599 third-year students enrolled in the mandatory 6-week family medicine clerkship over three years. Technical issues and timing were further monitored during the live study phase.

Scores from the closed book KA and open internet IMA were compared using paired Student's t-tests.

Additionally, exam scores were compared with clinical performance grades rendered by preceptors (family physicians hosting students in their clinical practices). Preceptors assess students using Likert (1-4) ratings of domains such as medical knowledge and self-directed learning, and also select an overall clinical grade from five levels: honors, high pass, pass, low pass and fail.

In the present study, the open internet IMA tests competency, or the achievement of a minimum skill level in a criterion-referenced (not normative) manner. Students earning higher clinical performance grades exceed expectations, reaching well above minimum required competency. Therefore, to capture data on students not meeting competency, clinical grades–grades rendered by preceptors based on clinical performance, not on exam score–in the lowest three categories (pass, low pass and fail) were compared with IMA performance. 

In this low-performing group, preceptors’ Likert scale (1-4) ratings of student skills in information mastery and self-directed learning (skills related to self recognition of need to access point-of-care information) were also analyzed.

This study was approved by the Institutional Review Board of Tufts Medical Center. 

Regarding feasibility, the exam was successfully administered to 599 students over 3 years. Over 90% of students completed the exam in the allotted time, and all demonstrated ability to use web resources to answer clinical questions. Cognitive interview data on reactions to the open internet IMA revealed student concern about time pressure, but otherwise showed an appreciation of the higher-order skills tested (Table 3).

Regarding performance, the open internet IMA mean score of 81% was not significantly different from the closed book KA mean score of 82% (t-value 0.47, NS). Half of students earning poor clinical grades from preceptors scored below average on the closed book KA, but two-thirds of these students scored below average on the open internet IMA. Within this group, preceptors gave 95% of these low-performing students poor Likert (1-2 of 4) ratings specifically in the subsection area of information mastery and 65% poor ratings in self-directed learning. In addition to their low quantitative (Likert) ratings in these two domains, preceptors’ written descriptions of low-performing examinees commented on poor synthesis and application of knowledge, and weak self-directed learning skills.

An open internet family medicine clerkship final exam testing application of knowledge and information mastery skills is feasible. Unlike in prior literature on closed book versus open resource exams,³ students in the present study performed as well on open internet competency-based exam questions as on closed book knowledge-based questions. They also appreciated the higher-order skills and simulation of clinical practice. Notably, the comment that this exam is “fun to take” is not typically heard after traditional exams.

IMA scores and clinical grades were correlated; both assessment tools identified low performers. By observing students in the clinical setting over 6 weeks, preceptors frequently identified students lacking higher-order knowledge synthesis and application skills; likewise, the IMA detected the same students. 

One study weakness is the need for detailed exam validation. Further validity research will use item analysis and comparison of high versus low performers. While it would be straightforward to compare this IMA with standardized tests such as FM Cases or NBME “shelf” exam, the latter are knowledge assessments. Better competency-based comparators would be clinical performance scores across all required clerkships, high stakes Objective Structured Clinical Exam (OSCE) scores, or residency director evaluations of medical school graduates after PGY-1.

As in most education research, the most compelling outcomes are the hardest to measure. Although students endorsed learning from the IMA, it remains unknown whether the exam corresponds with long-term retention. If validated, this kind of exam should be used more broadly to assess critical skills not currently tested in other parts of the curriculum.

The present findings are important because in an age of internet-based knowledge evolution, physicians and medical trainees must be competent at quickly accessing, synthesizing, and applying ever-updating information to make clinical decisions. Some argue that today’s information explosion mandates that evaluation of physicians’ practice include assessment of this ability, which can be done via open resource exams.³ Some specialty boards are considering changing recertification assessment methods to better align with higher-order skills needed in clinical medicine. Likewise in medical schools, an exam such as the present IMA might even replace knowledge-based tests. Because assessment drives learning,¹¹ we should not only discuss concepts of information mastery with our learners, but also assess their skills in providing quality patient care based on the most current information available. After all, as we tell our students, life itself is open book.