A Very Short List of Common Pitfalls in Research Design, Data Analysis, and Reporting

Performing scientific research without falling victim to one of the many research design, analysis, and reporting pitfalls can be challenging. As a medical statistician with research experience in a variety of medical disciplines, I regularly come across (and sometimes have been the cause of) avoidable errors and inaccuracies. Without such errors, research would, at the very least, be more informative to the readership of the research manuscript. In this article I present a short, nonexhaustive list of issues to consider.

As the starting point of all scientific endeavors, it is incontrovertibly important to clearly define the research questions and aims. The subsequent planning of the collection of useful data and formulating adequate statistical analysis often becomes easier once it is clarified whether the ultimate aim is to predict, explain, or describe.¹ If the ultimate aim is to explain, the ideal design is often an experiment (eg, a randomized controlled trial). Conversely, for many health-related research questions, nonexperimental data are the only viable source of information. This type of data is subject to factors that hamper our ability to distinguish between true causes of outcomes and mere correlations. For instance, for a nonexperimental before-after study, a change in the health for some individuals over time is easily mistaken as evidence for the effectiveness of a particular curative treatment, which may just be caused by regression to the mean.² To avoid such errors, studies with an explanatory aim may benefit from applying causal inference methodology.³

A too-small-for-purpose sample size may result in overfitting,⁴ imprecision, and lack of power, which can ruin a study of any kind. It is worthwhile to calculate the minimal sample size required to avoid disappointment.⁵ It is usually wise to be skeptical about rules of thumb for sample size.⁶

After data have been collected and cleaned, and initial data analysis⁷ has been completed, it often requires a large amount of self-discipline to follow the a priori defined statistical analyses plan (if one is even available). Indeed, it is hard not to look at every potential association in any given data set and even harder to unsee what is possibly just a false positive once one has started data dredging.⁸

After data collection, some researchers seem to have the natural tendency to immediately dichotomize measurements that were originally measured on a continuous or ordinal scale, such as dichotomizing the age of patients into groups of young and old. This natural tendency to dichotomize, sometimes referred to as dichotomania,⁹ is very often a bad idea.¹⁰ Various approaches exist that allow for data to be analyzed and made insightful on a continuous scale.¹¹

The presence of measurement and misclassification errors in data sets (present in most data sets, in my experience) are often wrongfully considered relatively unimportant.¹² Some have even argued that only the strongest effects will be detected in data that contain measurement error.¹³ This misconception that only the strongest effects will survive, I call the noisy data fallacy. Many statistical approaches exist that account for measurement and misclassification errors.¹⁴

Likewise, some degree of missing data is almost unavoidable in any study. Methods to deal with missing data, such as multiple imputation,¹⁵ have been criticized for making strong, untestable assumptions. While this is true, what is easily forgotten is that the assumptions made when ignoring missing data are often even stronger.¹⁶

Data are also often clustered. That is, data are often obtained from multiple centers, multiple studies, or multiple measurements within the same individual (eg, time series). In these settings where some data are more alike than others, it is often important to adjust the analyses accordingly.¹⁷

While many readers are quick to point out that a statistically significant effect does not mean the effect is also large enough to be relevant, it seems easier to forget that effects that are not statistically significant may not carry strong evidence that the effect does not exist.¹⁸ Contrary to popular opinion, removing variables that are not statistically significant from the analysis may not improve interpretation¹⁹ and may increase the chances of overfitting.²⁰

Given the many pitfalls in interpretation of P values and statistical (in)significance,²¹ some researchers—and even scientific journals—have called for the abandoning of statistical significance.²² It may then be tempting to ignore all uncertainty in statistical analyses and base conclusions solely on the value of a single-point estimate (eg, regression coefficient). Such point-estimate-is-the-effect-ism²³ relies heavily on the assumption that the point estimate is a valid and precise estimate of the true value, which it often is not.

One of the keys to success for valid causal inference in nonexperimental data is the adequate handling of confounding.²⁴ Successful adjustment for confounding means being able to distinguish potential confounders from intermediates in the causal chain between the factor of interest and the outcome²⁵ and colliders,²⁶ which sometimes is more easily said than done.²⁷ If the right confounders have been selected and adjusted for through, eg, by multivariable regression analysis (notice the distinction from multivariate regression²⁸), it is tempting to also interpret the regression coefficients of the confounding variables as being corrected for confounding, which would be committing a common error known as the Table 2 fallacy.²⁹ While substantiating causal claims is often difficult, avoiding causal inference altogether or simply replacing words like “cause” by “association” is not often the solution.³⁰

With the increasing use of machine learning and artificial intelligence in health care research, this incomplete list of common research design and analysis pitfalls may seem somewhat old-fashioned. Despite the arguably more complex nature of such analyses, many of the aforementioned issues also apply to such studies.³¹ Among all pitfalls mentioned, the easiest pitfall to avoid is that of incomplete reporting. Avoiding that type of error can be done simply enough by using reporting guidelines (see https://www.equator-network.org/).