Misalignment of Biostatistics Content Between Licensing Exam Study Aids and Contemporary Medical Research

Statistical literacy is a key skill set developed during medical training that facilitates physicians’ evidence-based clinical reasoning and application of scientific literature. ^1-3 Physicians’ application of biostatistics knowledge supports their ability to interact knowledgably with the literature (including both primary research and evidence-based guidelines) and to make evidence-based decisions regarding the care of their patients. Primary care represents many patients’ sole contact with medical professionals, so for family physicians to be able to effectively interpret clinical research to help their patients make informed decisions regarding their health is especially important.

Medical students now commonly complete undergraduate biostatistics coursework to develop these skills prior to medical school matriculation, ⁴ and preclinical curricula now increasingly emphasize exposure to concepts of evidence-based medicine concepts. ^5-10 In addition to the clinical relevance of comprehensive biostatistics training, biostatistics and epidemiology are core components of national licensing examinations for medical trainees. ^{11, 12} Given the importance of licensing exam performance for residency placement, many medical students supplement formal curricular content with self-guided commercial resources, including texts, videos, question banks, and preparatory courses. ^{13, 14}

Despite exposure to biostatistical content in the undergraduate medical education curriculum and in cocurricular commercial study resources, current medical students and practicing physicians continue to express difficulty with interpreting statistics in the clinical literature, especially in studies that use complex or novel statistical methods. ^15-17 This challenge has been exacerbated by the increased complexity of statistical methods used in the medical literature, creating a mismatch between advancing biostatistical methods in medical research and the relatively static content of study resources and curricula, ^18-21 and prompting calls for enhanced biostatistics training in medical education. ^{18, 22} The ubiquity of commercial study resources for licensing exams means that medical students frequently are exposed to biostatistics content through this medium, so understanding whether biostatistics knowledge acquired from exam study aids would plausibly prepare students to interpret biostatistical methods in contemporary medical research is important. Moreover, analysis of the biostatistics content in these resources could reveal potential deficits in statistics education and inform future redesigns of curricula and study materials.

In this study, we aimed to compare statistical methods and concepts employed in contemporary biomedical literature to biostatistical topics covered in commercial study resources used to prepare for licensing examinations taken by medical students in the United States—specifically, United States Medical Licensing Examination (USMLE) Step 1 and COMLEX-USA Level 1. We hypothesized that a significant discrepancy exists between the biostatistical methods described in board study materials and those used in contemporary medical literature, including (a) an emphasis in licensing exam study materials on biostatistics concepts that are rarely encountered in the biomedical literature, and (b) a lack of coverage of other biostatistics concepts that are frequently encountered in contemporary biomedical research.

This was a bibliographic study not involving human research subjects and therefore did not require institutional review board approval. Following similar content analysis studies, ^{18, 22} we sampled issues of the Journal of the American Medical Association (JAMA), the British Medical Journal (BMJ), and the New England Journal of Medicine (NEJM) using a stratified random sampling approach. Based on weekly issues published during 2023, we randomly selected two issues per month per journal, yielding a total of 72 issues. Within each issue, we selected articles for review if they involved primary or secondary data analysis, including original research (clinical or basic science), brief reports or research letters, and systematic reviews with meta-analyses. We excluded letters (excepting research letters reporting original data analysis), images, commentaries, case reports, editorials, treatment guidelines, and any other article type not reporting results from original data analysis. We also excluded theme issues; if a theme issue was selected randomly, we used the subsequent available issue.

We determined a reading schedule a priori by randomly sorting issue and article order, and then reviewed each article for statistical content. We recorded each discrete statistical method (eg, linear regression model) or concept (eg, P value) described in the methods section of the main article text. Although biostatistics often are taught to medical students alongside epidemiological and other concepts, we focused our study on statistical methods only, operationalized as methods used to summarize, compare, or interpret quantitative data. For each article, we recorded the presence of each statistical method or concept as a binary variable. To establish interrater agreement, two authors read and analyzed the first 20 articles. Thereafter, a single author reviewed all remaining articles and extracted key words or phrases that were used in the final analysis. Any coding discrepancies in the first 20 articles and any ambiguous coding of subsequent articles were resolved via discussion and consensus.

We selected USMLE Step 1 and COMLEX-USA Level 1 commercial study aids based on medical students’ documented use of these resources and positive correlation with licensing exam scores. ^3-6 To incorporate a wide range of affordable materials corresponding to varied study modalities, we chose one comprehensive review text, one online biostatistics-specific question bank, and an online review video depository, all updated in 2023. Given the relatively succinct nature of these materials, one author reviewed the entirety of each resource’s biostatistics content.

After completing data extraction, we created a unified list of all discrete methods or concepts (key words or phrases) extracted from the sampled articles and study resources. Two authors independently generated a list of domains encompassing these methods and concepts, and they arrived at a final domain list after discussion. The two authors then independently mapped each method or concept mentioned in the articles and study resources to one of the domains, and they arrived at a final classification after discussion of discrepant domain coding. For each domain, we determined (a) whether it was present or absent in each study resource; (b) whether it was present or absent in each journal article; and (c) in how many articles it was present out of the total number of sampled articles (frequency). A priori, we defined “frequently occurring” concepts in the biomedical literature as categories in the top quartile of frequency and “infrequently occurring” categories as those in the bottom quartile of frequency. Complete data and replication code for our analyses are included as Supplemental Files 1 and 2.

We identified 273 eligible journal articles (124 in JAMA, 53 in BMJ, and 96 in NEJM) and extracted 4,728 key words from these articles and from the three study resources. After excluding 776 key words determined not to represent a specific statistical method (eg, excluding the phrase “sensitivity analysis,” which could refer to any method or test depending on the context) and removing duplicates in the remaining key words, we retained 1,057 unique key words (methods or concepts) for further classification into domains of biostatistical methods. We summarized the biostatistical methods and concepts into 19 domains and one residual category of statistical methods not elsewhere classified (Table 1).

We examined coverage of each domain within the three board exam study resources (Table 2). These study resources lacked any coverage of methods related to comparing nonnormally distributed data between groups, fitting generalized linear regression or multilevel regression models, or using weights in statistical analysis. Additionally, the board exam study resources did not always cover concepts related to linear regression, survival analysis, meta-analysis, or handling of missing data. Importantly, Table 2 represents coverage of any concepts within each domain but does not address the depth of this coverage. For example, all study resources were credited with covering the “statistical significance and multiple testing” domain because all three resources addressed P values, α levels, and hypothesis testing, but none of these resources went so far as to discuss correction of significance testing for multiple comparisons.

We compared the frequency of each domain within the 273 journal articles included in our analysis (Table 3). High-frequency domains included statistical error, variance, and confidence intervals; statistical significance and multiple testing; measures of central tendency; power analysis, sample size calculation, and stopping criteria; and group comparisons of categorical data. Low-frequency domains included correlation of continuous data; group comparison of nonnormally distributed continuous data; diagnostic test and predictive score performance; meta-analysis methods; and weighting. Considering the four domains that received no coverage in any of the three board exam study resources (as shown in Table 3), we found that 172 of the 273 (63%) articles included methods from at least one of these domains.

The accelerating pace of discovery in biomedical research highlights the importance of interpreting original research in light of emerging diseases, new therapeutic options, and changes in population health. ^{18, 19} Primary care physicians are uniquely positioned to interpret and apply the findings of biomedical research because they are often patients’ primary or even only point of contact within the medical system. Despite the availability of evidence-based point-of-care resources synthesizing the research literature for practical use, the many instances of rapidly evolving science, conflicting evidence, or conflicting interpretations of the same evidence should prompt physicians to examine the methods and findings of the primary literature more closely. In this study, we demonstrated that a major discrepancy persists between biostatistical methods frequently occurring in clinical research and the methods that are emphasized in medical licensing exam study resources, potentially contributing to significant learning gaps among physicians in training. Our study thus reinforces prior studies’ ²² calls for overhauling statistics curricula in undergraduate medical education and provides initial data regarding specific statistical methods to be incorporated in revised curricula.

One-fifth of all domains of biostatistical methods were omitted across all three study resources, including multilevel regression modeling and data weighting. Notably, all but one of the domains (ie, generalized linear regression) defined in our study were represented in the most recent USMLE and COMLEX-USA content outlines, which encompass expected knowledge across all three standardized licensing exams. ^{11, 12} All high-frequency domains noted in the journal articles were included in all study resources, but their coverage was often superficial relative to the articles’ level of detail. The complexity of a statistical method (from the reader’s point of view) is based on the level of prior knowledge assumed when describing the method, the computational difficulty of implementing that particular method, and the relevance of that method to the reader’s area of practice, with different statistical methods being more or less common in different specialties. Our coding scheme assumed that if a student learns about statistical power, for example, they also understand interim analyses and stopping criteria derived from power analyses. However, incomplete coverage of these concepts in study resources can contribute to medical graduates’ limited ability ^23-26 and overestimation ^{27, 28} of their skills to use statistics to make evidence-based decisions. While all study resources emphasized concepts relating to diagnostic testing and predictive score performance (eg, sensitivity, specificity), this major component of licensing exam preparation rarely was encountered in the journal articles, though we acknowledge its distinct use in clinical practice. ^{29, 30} Nonetheless, even with perfect understanding and retention of biostatistics content encountered on licensing exams, medical trainees may not be prepared to interpret contemporary statistical methodology in biomedical research. ³¹

Prior studies in medical education have indicated a need for improved statistical training, ³² and our study underscores the need for enhancing education in this area. Given the breadth of statistical concepts, expecting medical students to become familiar with every methodological term that might appear in primary research literature is unrealistic; rather, educators must find effective frameworks for maximizing statistical learning of core concepts and frequently used techniques during the preclinical and clinical years. ³³ Within medical school curricula, enhanced longitudinal exposure with spaced repetition may be the first step, along with journal clubs and research review during case presentations. ⁷ Looking ahead to these interventions in the clinical years, the foundational curriculum also can signpost statistical concepts that likely will remain relevant to students’ education and practice beyond the period of preparing for licensing exams. Faculty development to improve the teaching of statistics can be supplemented by hiring biostatistician educators, although this may be difficult at schools without affiliated biostatistics departments. ⁹ Study resource revision may be the most challenging area to reform directly due to the independence of third-party companies producing study resource content; but faculty- and student-led efforts to create study materials can be scaled up to supplement commercial resources. ³⁴ Regardless of what changes medical educators ultimately make, intentionality and persistence are prerequisites for improving biostatistical education.

The present study is limited in several regards. Analysis of commercial study resources may not reflect the content of medical school curricula, which vary across institutions. Furthermore, the authors’ professional background and experience with statistical analysis influenced the development of the coding scheme, and other groups may prefer more granular schemes or different classifications of specific terminology within the domains presented in our study. The three commercial study resources were chosen for their varied modalities and students’ documented preferences, but they may not be fully representative of all available study aids for licensing exams. Similarly, we chose to sample articles from three high-impact journals selected for their wide readership and reputation, but our findings may not be generalizable to all recent biomedical research, especially because we sampled articles from only 1 year. Moreover, we extracted data on statistical concepts from the methods section of each published article, but did not review the results, tables, figures, or supplemental content, which may have contained additional statistical terminology not mentioned in the methods section.

Overall, our comparison of biostatistics content in medical licensing exam study resources and three major biomedical journals indicates that medical trainees may be critically underprepared to interact with contemporary clinical research because of incomplete coverage of statistical methods in materials aimed at preparing them for standardized tests. Limited longitudinal training in biostatistics methods can restrict students’ ability to critically evaluate both primary research and point-of-care resources that synthesize the existing literature, as well as potentially limit their ability to contribute to primary care research once they are in practice. In addition to testing the efficacy of educational interventions to close this gap, future studies could incorporate content analysis of medical school statistical curricula; expand the sample size and date range of sampled articles and study resources; examine how study resources are matched or mismatched with biostatistical methods that are most common in particular specialties; or conduct longitudinal analyses of multiple editions of study resources to assess for possible evolution of statistical content. We must ensure that medical students and physicians become better trained to recognize and interpret the wide array of statistical methodologies that legitimize or invalidate the novel diagnostic and treatment options offered to patients.

This article’s content was presented in poster form at the Brody School of Medicine Distinction Day (Greenville, NC; April 11, 2024) and in a brief podium presentation at the Brody School of Medicine Medical Education Day (Greenville, NC; April 25, 2024).

D.T. discloses salary support from Kate B. Reynolds Charitable Trust and Lilly Grant Office for research and quality improvement projects unrelated to this work. The authors have no other competing interests to declare.