“Allegedly”

Working_in_the_dark_ViaErnstGräfenberg

“Allegedly” is one of those words that people stick in front of disputed things, and it serves the useful purpose of signaling that the dispute exists. But there is another way people use it as well, and that is less about signaling dispute and more about introducing it. And it works! For me, as a reader, when I see the word “alleged” tied to something, it makes me more critical, more doubtful, and more aware that some other people don’t think the thing in question is true.

So, I find it rather disturbing when people use the word “alleged” for things like sexual assault, abuse, and online harassment. In this context, the word is used as a rhetorical trick, even (especially?) when the event itself is not really in doubt, to create that doubt. People use this word, in short, to minimize the experiences of women.

Continue reading

Brutal Honesty

treeswallows_viaAnitaRitenourHonesty is the most important thing—at least, that was a value I learned growing up. No criticism was left unspoken, nor was there any thought that it should be. I learned to value blunt, direct language. I learned to say what I thought. I learned to be brutally honest, and to believe it was the right thing to do.

What I learned was not unique. I see a lot of people who prefer to be direct and who find honesty refreshing. I know a lot of people who find subterfuge and subtext exhausting, and who are actively annoyed by people who weave and bob and refuse to say what they think. And I, like a lot of other people, am actively annoyed by the fact the public figures say whatever they think people want to hear with no regard for truth. Honesty, I think, is objectively valuable.

What I didn’t learn, at least for a while, is that brutality and honesty need not go together.

Continue reading

True Stories

books_viaxlibberPerhaps I am too much a student of Tim O’Brien, but I believe that the purpose of stories, literal or otherwise, is to contain truth. In his magnificent opus, The Things They Carried, O’Brien gave us the truth of being a soldier during the Vietnam War. It did not much matter that many of the things in his book were not literal events, because they contained the rich truth of that experience. Stories shine when they convey a truth of experience too big for simple events.

Which is not to say events do not matter. There is another, related role for stories: to provide context for the world in which we live. They are foils for everything we see and experience, catalogues of sensation and emotion, especially and personally constructed to anchor us on deep, shifting sands. So we would like our stories to feel true, in Tim O’Brien’s sense, and also be true, in a more literal sense. And yet, we also conflate those two.

Continue reading

A More Perfect Truth

credit: Flickr user shawncalhoun

The scientific method, at its heart, is a set of steps to keep us from fooling ourselves, and one another, and thus to arrive at our best approximation of truth. Each step in the traditional scientific method is a way we reduce bias, eliminate confusion, and further our collective knowledge. But recent high profile research has highlighted some of the ways it can break down along the way, especially in preliminary research.

The idea that preliminary research is mostly inconclusive or incorrect isn’t surprising—preliminary studies are the way a scientific community investigates new ideas. Contrary to public perception, publication in a scientific journal is not so much verification of truth as it is the beginning of a debate. Collective knowledge, in theory, builds from that point onward.

So, when I read recently that more than two-thirds of a group of psychological studies could not be replicated, I wasn’t too surprised. Whatever the media might make of a single small study, and however much they might tout it as a breakthrough (and they do, for everything), the chances are that the results are flawed somehow. Scientists, of course, are still human, and they still get pulled toward positive data. There are a number of habits, like abusing the P value (try it yourself to see how it works) or choosing what measures to focus on after the fact, that can lead to a researcher misrepresenting results, even unintentionally. And, of course, there are a few bad actors who inflate the results of their studies on purpose.

There is a secondary problem in science as well, which is that journals tend to publish positive studies, and researchers tend not to even submit negative studies, leading to publication bias. If you’re a drug company, you might abuse publication bias on purpose to make your products look more effective than they actually are. To makes things worse, journals have their own bias towards new research and often don’t want to publish negative studies or failed replications of previous research. Combined with the set of problems I mentioned above, which lead to iffy research, publication bias effectively hobbles scientific debate by letting lots of ideas in, but silencing the voices that would weed out the bad ones.

You might have noticed that the first set of problems arises from individual biases, while the second set arises from systemic biases. In the first case, researchers are accidentally or intentionally allowing bias into their studies and tainting their results. The scientific method is still subject to human error, or intentional gaming of the system. In the second case, the scientific method has few tools for eliminating systemic biases, so a slightly more developed solution is needed. The only current tool is peer review, but that has its own host of limitations and problems.

I think, however, there is a solution that would reduce problems at both levels simultaneously, and it’s one we already know works: pre-registering research.

Pre-registration of clinical trials is a tool recently employed to deal with publication bias in medicine, and especially to prevent bad actors (such as drug companies with a financial stake in the matter) from gaming the system by hiding negative research. It also eliminates researcher biases because they have to register their methodology before conducting the study, and thus cannot modify that methodology during or after the fact to generate a positive result. The effect has been a dramatic decline in false-positive results.

Some people have rightly pointed out the problems with pre-registering all research, and how difficult it would be to figure out who to register with and how to keep track. This is where the second part of the solution comes in: journals already admit that there is value in publishing negative results, so register prospective research methodologies with scientific journals, which in turn must commit to publishing the end result. Even if that commitment came with some caveats, this would simultaneously prevent researchers from modifying their methodology, thus reducing biased results, and force journals to accept research based on the methodological merits when they are still blind to the outcomes, thus reducing biased publication.

Of course this wouldn’t solve every potential problem in science, but, as I said, science is not a perfect enterprise—it is a collective endeavor to arrive at the best approximation of truth. We can always do better, and we can always learn more, and it’s time to take the next step in that direction. We know we need to eliminate our individual biases, and now we know that we need to address collective biases as well. We also know that we can—it only remains to do so.

Image credit: Flickr user shawncalhoun