Caveat Lector

In 2019, Cold Spring Harbor Laboratory, Yale University, and BMJ Publishing Group established medRxiv, an internet server with the stated purpose of allowing scientists to share and receive feedback on their work prior to publication.1 As the COVID pandemic spread across the globe and demand exploded for information about the virus and possible treatments of the resulting disease, the site was indexed by PubMed, in February 2020. medRxiv is not the only website that offers access to unpublished scientific papers prior to peer review, and it is not the only such site indexed by PubMed. In June 2020, the National Institutes of Health (NIH) launched the NIH Preprint Pilot, opening PubMed to papers from approved preprint sites.2 If you are not familiar with this phenomena, two questions probably come to mind: why is online publication of scientific articles prior to peer review becoming more prevalent, and is this a good idea? The “why” question is easy to answer. Criticism of scientific peer review is nothing new. Formal peer review often takes over a year from submission to publication and this delay slows the progress of science in ways that can be more than just inconvenient. There is also long-standing concern about the fairness of the peer-review process, particularly its tendency to resist new ideas and methods in favor of orthodoxy. Over the past decade, the publishing world has transformed from its traditional focus on print media to electronic communication. We all know how this has impacted newspapers and magazines. Its effect on scientific journals in general, and medical journals in particular, is no less dramatic. Particularly during the pandemic, the public is easily frustrated by the slow pace of science, and misinformation fills the void while we wait for valid evidence to emerge.

Whether or not online publication before peer review is a good idea is a harder question to answer. Who reads these papers? Is their dissemination limited to the scientific community or do they reach the general public? How many of them are eventually published in peer-reviewed journals? Most importantly, does the resulting information turn out to be true?

In this issue of Family Medicine, LaKesha Anderson, PhD, and Christy Ledford, PhD, examine the ramifications of online publication of scientific articles prior to peer review in the wake of the COVID-19 pandemic.3 They examined preprint articles posted on medRxiv between February 15, 2020 and May 22, 2020, focusing their study on 39 articles addressing two potential COVID-19 treatments. Their outcome variables were whether the papers were later accepted by a peer-reviewed journal, whether they were eventually withdrawn from medRxiv, and three different measures to estimate the number of people exposed to each article. Exposure was not limited to counting web page hits for each paper; it also included citations in lay news outlets or social media. Their results indicate that these papers had wide impact outside of the scientific community. Two of the 39 papers were later withdrawn by the authors and only five of the 39 were eventually published in the peer-reviewed literature. They conclude that medRxiv disseminated new information about COVID treatments with both positive and negative findings, and this information reached the general public before being scientifically scrutinized. Remarkably, there was more social media exposure for the two withdrawn papers than for the five papers later published in peer-reviewed journals.

There are several important take-home messages in Drs Anderson and Ledford’s paper. First, finding a paper on PubMed no longer guarantees that the paper has undergone scrutiny by peer review. It has always been challenging to teach family physicians how to evaluate new research evidence. Electronic publication before peer review makes this harder. Second, many of these papers were cited in the lay press and on social media, often accompanied by whatever interpretation best suited those citing them. Since the end of pharmaceutical marketing restrictions in 1981, family physicians have become accustomed to questions about drug treatments from our patients. With 40 years of experience, we have learned not to trust marketing from drug companies, and have taken steps to restrict access to it by our learners.5,6 Are there any family physicians in America who did not field questions from patients about hydroxychloroquine as a treatment for COVID-19? Who conducted the studies of this medication that were published online without peer review, and who funded their work?

At the start of the Mueller Investigation of the 2016 presidential election, David Roberts wrote that America is facing an epistemic crisis.7 He defined this as a breach in our shared societal consensus regarding what evidence is sufficient to prove something is true. The idea of an epistemic crisis dates back to the work of philosopher Alisdair MacIntyre in 1977.8 In traditional journalism, sufficient evidence of truth is based on the agreement of two independent sources of information. This is what editors expect of reporters before stories are published in a reputable news source. In science, sufficient evidence is based on peer review, a process in which two or more experts confirm that a given experiment or observation is based on valid research methods, analysis, and interpretation. Family Medicine is a peer-reviewed medical journal that is listed in the National Library of Medicine’s Index Medicus. We are required to maintain clear standards to retain this recognition, but the standards boil down to one basic principle: the journal assumes responsibility to print evidence that is true and to reject evidence that is not proven to a sufficiently high standard. We exercise this responsibility by organizing a community of scholars who agree to review one another’s work anonymously as a service to our discipline in the public interest. Peer review is not easy, and it is not infallible. But peer review is the epistemological foundation standing between authors and readers of scientific papers. Publishing science before peer review removes this foundation, replacing it with the independent judgement of each reader.

Perhaps it is not our place to criticize PubMed. After all, they are conducting a pilot study of preprint publication and the policy is not yet set in stone. Maybe the roots of this phenomenon lie within the research community itself. Most of us work at universities that encourage faculty interviews with the lay press about emerging science. It is flattering to be asked to do these interviews and both individual faculty members and their universities relish opportunities to enhance their reputations. But one could argue that the horse is already out of the barn when these interviews are published in the local newspaper or broadcast on the radio. Maybe PubMed is just trying to catch up with a dissemination process that is already beyond quality control. We struggle to distinguish between marketing and science even as marketing has inched its way into our own universities. Marketing is selling. Its guiding principle is contractual, with the motto of caveat emptor: let the buyer beware. Science is about rigorous proof with evidence, and its guiding principle is a moral covenant to seek truth in the public interest. There is a lot at stake in the debate about publication before peer review. There are plenty of people who would not mind seeing the evidentiary standard of science lowered in the interest of fast access and free speech. If caveat emptor works for sales, perhaps caveat lector—let the reader beware—will work for science journalism.

As an academic discipline, family medicine has always been more focused on clinical teaching than on research. Thus, we are more often the consumers of new evidence than we are the producers of it. Lowering the evidentiary standard for disseminating new science will make our work harder as physicians and as teachers and will hurt more people than it helps. In his 2017 article, Roberts warned that America is devolving into a “tribal epistemology” in which we conflate truth with whatever is good for our own tribe of believers. Those who wanted to promote hydroxychloroquine accepted the same evidence that others reject purely on the basis of their preexisting beliefs. Anderson and Ledford conclude their paper by stating, “Thus, family medicine educators need to modify how they teach evidence-based medicine in this evolving landscape.” How we teach will be the least of our problems in a world of caveat lector. Electronic publication before peer review is not in the public interest, and it is our responsibility to say so.