The scientific method, at its heart, is a set of steps to keep us from fooling ourselves, and one another, and thus to arrive at our best approximation of truth. Each step in the traditional scientific method is a way we reduce bias, eliminate confusion, and further our collective knowledge. But recent high profile research has highlighted some of the ways it can break down along the way, especially in preliminary research.

The idea that preliminary research is mostly inconclusive or incorrect isn’t surprising—preliminary studies are the way a scientific community investigates new ideas. Contrary to public perception, publication in a scientific journal is not so much verification of truth as it is the beginning of a debate. Collective knowledge, in theory, builds from that point onward.

So, when I read recently that more than two-thirds of a group of psychological studies could not be replicated, I wasn’t too surprised. Whatever the media might make of a single small study, and however much they might tout it as a breakthrough (and they do, for everything), the chances are that the results are flawed somehow. Scientists, of course, are still human, and they still get pulled toward positive data. There are a number of habits, like abusing the P value (try it yourself to see how it works) or choosing what measures to focus on after the fact, that can lead to a researcher misrepresenting results, even unintentionally. And, of course, there are a few bad actors who inflate the results of their studies on purpose.

There is a secondary problem in science as well, which is that journals tend to publish positive studies, and researchers tend not to even submit negative studies, leading to publication bias. If you’re a drug company, you might abuse publication bias on purpose to make your products look more effective than they actually are. To makes things worse, journals have their own bias towards new research and often don’t want to publish negative studies or failed replications of previous research. Combined with the set of problems I mentioned above, which lead to iffy research, publication bias effectively hobbles scientific debate by letting lots of ideas in, but silencing the voices that would weed out the bad ones.

You might have noticed that the first set of problems arises from individual biases, while the second set arises from systemic biases. In the first case, researchers are accidentally or intentionally allowing bias into their studies and tainting their results. The scientific method is still subject to human error, or intentional gaming of the system. In the second case, the scientific method has few tools for eliminating systemic biases, so a slightly more developed solution is needed. The only current tool is peer review, but that has its own host of limitations and problems.

I think, however, there is a solution that would reduce problems at both levels simultaneously, and it’s one we already know works: pre-registering research.

Pre-registration of clinical trials is a tool recently employed to deal with publication bias in medicine, and especially to prevent bad actors (such as drug companies with a financial stake in the matter) from gaming the system by hiding negative research. It also eliminates researcher biases because they have to register their methodology before conducting the study, and thus cannot modify that methodology during or after the fact to generate a positive result. The effect has been a dramatic decline in false-positive results.

Some people have rightly pointed out the problems with pre-registering all research, and how difficult it would be to figure out who to register with and how to keep track. This is where the second part of the solution comes in: journals already admit that there is value in publishing negative results, so register prospective research methodologies with scientific journals, which in turn must commit to publishing the end result. Even if that commitment came with some caveats, this would simultaneously prevent researchers from modifying their methodology, thus reducing biased results, and force journals to accept research based on the methodological merits when they are still blind to the outcomes, thus reducing biased publication.

Of course this wouldn’t solve every potential problem in science, but, as I said, science is not a perfect enterprise—it is a collective endeavor to arrive at the best approximation of truth. We can always do better, and we can always learn more, and it’s time to take the next step in that direction. We know we need to eliminate our individual biases, and now we know that we need to address collective biases as well. We also know that we can—it only remains to do so.

Image credit: Flickr user shawncalhoun