“When you hear a scientist talk about ‘peer review’ you should reach for your Browning.”

                                 James Delingpole, writing for Breitbart, 28 October 2016

We should all hope this is an extreme view.  Most policy makers demand that any science that affects their decisions has been subjected to peer review.   Peer review provides the Good Housekeeping seal of approval for science consumed by the general public.  Problem is:  how many of us really know what peer review is and whether or not it can eliminate erroneous and fraudulent science from contaminating what we generally accept as the truth.

Peer review begins when a scientist finishes a study and submits a manuscript to a journal, where the editor typically sends the paper to a couple of known experts in the subject area for their anonymous comments—a review of the work by peers.  These comments guide the editor’s decision to accept or reject the paper.  Most papers that are published have benefitted from this process because even a good manuscript is revised by the authors to correct errors, clear up ambiguities, and add citations to the work of others, as suggested by the reviewers.  Academics provide these reviews to the journals free-of-charge in the spirit of helping the progress of science and paying their dues as a member of the academic community.

If a paper is rejected, the authors can send it to another journal to begin the process again.  In most fields of study, there are a huge number of journals, so with modest persistence an author is likely to find an outlet for the work somewhere. 

Some journals have better reputations than others.  The most selective journals publish about ten percent of the manuscripts they receive.  Journals published by the major scientific societies generally garner greater respect than those managed by commercial publishers.  Increasingly I receive requests to submit work to obscure journals that seem to have no reputation at all.  Even so, a paper published in such a journal might be described as peer-reviewed. 

To ease the editor’s workload, some journals ask authors to suggest potential reviewers for their manuscript.  Only a fool would suggest anyone other than a good friend, and with the increasing flood of submitted manuscripts, it is extremely hard for editors to ferret out who might be a biased reviewer.   Authors who recommend their students, relatives, and even those who have been paid to provide a positive review are scheming the system to ensure positive peer review.  

Even appropriate reviewers are hard pressed to detect all errors in scientific work. Few reviewers have the time to check all the calculations in a manuscript or to reanalyze the statistics to see if they are robust.   And reviewers must assume that the reported data are real; it is nearly impossible to detect fraud in a manuscript without redoing the work in an independent laboratory.  Scientific work that is erroneous or fraudulent may take years to be discovered by someone who has taken the time to repeat the study.

Reviewers who spend more than a day to offer comments on a manuscript will quickly find that they are falling behind in their own work.  Last year, I was asked to review 27 manuscripts for 16 different journals.  You try to do the best you can.   

By itself, the phrase published in a “peer-reviewed” journal doesn’t mean much anymore.  No one likes to be a snob, but increasingly I find that my reactions to published work are flavored by where the work has been published—the reputation of the author, the journal, and the likelihood that the editors selected appropriate reviewers. In the process I probably overlook some good work published in “second-tier” outlets, but like most scientists I need to have a way to drink selectively from the “fire-hose” of published work that crosses my desk each day.  It is easy to drown in the sea of second-class manuscripts, even if they claim to have been peer reviewed.

I have been in academics for nearly 50 years, so it is relatively easy for me to sort this out, even if I miss a few good papers in the process. But, evaluating a scientific paper beyond the imprimatur of “peer review” is nearly impossible for elected officials, their staff, members of the media, and policy advocates.  Peer review is not foolproof.  Scientists must speak out forcefully when they see the policy process subverted by what they know is untrue, even if it has found its way into print.   

Eventually good science trumps bad science.  Careful measurements and well-designed experiments lead to established facts, which lead to the truth about what to do with policy. Peer review starts that process. The alternative is no review at all.