Negative results are decreasing in scholarly papers

One of the side effects of our fear of mistakes is the discrediting of negative findings. On the few occasions when I played craps in a casino, I noted how poorly the other players at the table reacted when I bet the "don't pass" line - essentially, betting on the dice roller to fail - when the outcome of a roll was perfectly random and the expected payout was no different whether you played pass or don't pass.

The craps example demonstrates how dysfunctional trying to deny the negative is. As Edison said, "[Negative results are] just as valuable to me as positive results. I can never find the thing that does the job best until I find the ones that don't."

Given the above, reading the abstract of this 2012 paper was both unsurprising and somewhat discouraging. Entitled "Negative results are disappearing from most disciplines and countries," by Daniele Fanelli and published in the March 2012 issue of Scientometrics, the paper indicates a significant increase of scholarly papers reporting that their study results supported the stated hypothesis, rather than disproving it:

This study analysed over 4,600 papers published in all disciplines between 1990 and 2007, measuring the frequency of papers that, having declared to have “tested” a hypothesis, reported a positive support for it. The overall frequency of positive supports has grown by over 22% between 1990 and 2007....

Fanelli notes some fascinating cultural differences in reporting negative findings, and included these wise words of warning:

A system that disfavours negative results not only distorts the scientific literature directly, but might also discourage high-risk projects and pressure scientists to fabricate and falsify their data.

Yes indeed.

[Hat tip @Mangan150]

See some prior posts on negative data in research: "Free the Dark Data in Failed Scientific Experiments," "Web site offers scientists access to lessons from failed experiments."

Free the "dark data" from scientific failures

I recently stumbled across "Freeing the Dark Data of Failed Scientific Experiments," by Thomas Goetz in Wired magazine. I had read it years ago, and its thesis still sounds as fresh today as it did back in 2007:

What happens to all the research that doesn't yield a dramatic outcome — or, worse, the opposite of what researchers had hoped? It ends up stuffed in some lab drawer. The result is a vast body of squandered knowledge that represents a waste of resources and a drag on scientific progress. This information — call it dark data — must be set free.

While the usefulness of negative data is being recognized, there are still powerful forces, organizational and human, working against freeing our dark data:

More and more, research is funded by commercial entities, which deem any results proprietary. And even among fair-minded academics, the pressures of time, tender, and tenure can make openness an afterthought. If their research is successful, many academics guard their data like Gollum, wringing all the publication opportunities they can out of it over years. If the research doesn't pan out, there's a strong incentive to move on, ASAP, and a disincentive to linger in eddies that may not advance one's job prospects.

One of the publications cited by Goetz is still going strong: "The Journal of Negative Results in BioMedicine," edited by Bjorn Olsen of Harvard Medical School, has possibly the most delicious description of any journal ever:

Journal of Negative Results in BioMedicine is an open access, peer-reviewed, online journal that promotes a discussion of unexpected, controversial, provocative and/or negative results in the context of current tenets.

When will all the results of scientific research be released into the wild? How can we free up this unused resource?

Web site offers scientists access to lessons from failed experiments

The New York Times Bits blog profiled the website ResearchGate, which gathers lessons from failed scientific experiments and makes them available to other researchers:

“Science is very inefficient,” says Ijad Madisch, founder of a Web site called ResearchGate. “You try an experiment, fail, try again, fail, try again, it works. And what works is what you publish. All the data about failure is wasted.”

Begun in 2008, ResearchGate claims to have 11,000 research and educational institutions among its users. It aims to be a place where people can share what they learned in the failed experiments. Some of this is documented, but a great deal more takes place across chat rooms where scientists informally exchange information.

The ResearchGate site also includes a user post asking for a database of data from failed experiments, a proposal I've seen before (but can't recall where). It's a great idea:

I think developing a database for the storage of failed experiments makes sense for experiments that lost the chance to be shown in published articles. Everyone could upload their experimental results which they find unsatisfactory, including the experiment method, experiment data, and the experiment purpose.

This experimental data can give a good experience and lessons to the researchers who want to perform similar experiments. They can improve their experiment based on the failing example. Someone might also give a different opinion for the unreasonable experimental data, and then test their idea. Maybe a novel theory can be developed from these failed experiments. Or, like the Post It, the unsuccessful experiment data could be used to solve other problems.