"My Bad!" Skillful research proves you need to own your mistakes to learn from them

There are a number of reasons I've written less on the blog this year. One is that, after more than 5 years of thinking about mistakes and writing the book, I was running out of interesting things to say and observe (it's not an accident that most of the posts this year are stories and not items discussing mistakes and failure).

But another important reason is what has been called the "fetishization of failure." It has just become too trendy and easy to write about how we should all learn from failure, "fail fast and often," etc. etc. Too easy to start conferences on the topic, Twitter feeds, etc.

I believe that most or all of the people who have jumped on the failure bandwagon recently are well-intentioned and wish to raise people's awareness of the values of failure. But the inevitable result has been a watering-down and trivialization of something I consider really important.

And then. And then, scholars with serious credentials in this area like Francesca Gino, Bradley Staats and Christopher Myers publish a paper entitled, "'My Bad!' How Internal Attribution and Ambiguity of Responsibility Affect Learning from Failure" - a deeply-researched, well-written, clearly presented argument on how individuals' ability to learn from failure is directly affected by how they view their responsibility for it.

This isn't a new idea (the book has a section on "owning your mistakes") but rarely have I seen as impressive a discussion of the topic - and new research to boot. Gino et. al. start off with as clear a summary as you can get of previous scholarship on learning from failure. For example:

Learning from failed experiences has been of particular interest as organizations seek to adapt and avoid repeating prior failures (Ingram & Baum, 1997, Kim & Miner, 2007, Madsen & Desai, 2010). Relative to successful experiences, failures have been seen as more effective triggers of individuals’ learning efforts, because they reveal a gap in ability that stimulates efforts to “tweak” existing practices, search for new capabilities, and develop innovative solutions (Sitkin, 1992, March & Simon, 1993, Baum & Dahlin, 2007, Hora & Klassen, 2013). Following traditional theories of individual learning (e.g., Kolb, 1984), failure can be seen as a form of unexpected event (i.e., where actual outcomes differ from expected outcomes; Allwood, 1984) that creates a sense of discomfort that triggers individuals to make sense of it, test hypotheses, and stimulate growth (Louis & Sutton, 1991, Ellis, Mendel, & Nir, 2006). At the same time, by revealing that an existing strategy is unsuccessful, failures encourage broader search for new strategies (i.e., exploration), resulting in enhanced long-term innovation (March, 1991, Audia & Goncalo, 2007). Indeed, after a successful experience, it is more difficult to detect deviations from a plan (Ellis, Mendel, & Nir, 2006), as the successful outcome confirms the validity of a prior routine (Weick, 1984, Sitkin, 1992) and builds confidence (and complacency) regarding its utility for future performance (Weick, Sutcliffe, & Obstfeld, 1999).

Note, first of all, the wealth of citations in the preceding paragraph. Learning from failure, or "unexpected events," which is closer to the definition I use in the book, is not a fad. It is a well-researched, grounded, fact. It is also easy to avoid - "ambiguity of responsibility" gives credence to individuals who wish to distance themselves from failure.

The new research presented is also profound. The authors test four hypotheses, two of which are central to everyone's ideas of learning from failure:

Hypothesis 2: The learning effects of failure are driven by internal attribution of the failed experience. Specifically, internal attribution moderates the effect of failure on learning, such that the effect of failure on learning is more positive when the failure is attributed more internally.

Hypothesis 3: Ambiguity of responsibility decreases internal attribution and learning from failure

The experiments carried out validated both these hypotheses - test subjects who took responsibility for a failure learned more, and situations where responsibility was unclear did not provide as strong a basis for learning from the failure. This is important stuff (and, as the authors suggest, an area where more study is needed).

Go ahead and follow the failure Twitter feeds and attend the conferences if you want, but, if you really want to understand how people learn from mistakes and how to improve the environment for such learning, read "My Bad!"

Negative results are decreasing in scholarly papers

One of the side effects of our fear of mistakes is the discrediting of negative findings. On the few occasions when I played craps in a casino, I noted how poorly the other players at the table reacted when I bet the "don't pass" line - essentially, betting on the dice roller to fail - when the outcome of a roll was perfectly random and the expected payout was no different whether you played pass or don't pass.

The craps example demonstrates how dysfunctional trying to deny the negative is. As Edison said, "[Negative results are] just as valuable to me as positive results. I can never find the thing that does the job best until I find the ones that don't."

Given the above, reading the abstract of this 2012 paper was both unsurprising and somewhat discouraging. Entitled "Negative results are disappearing from most disciplines and countries," by Daniele Fanelli and published in the March 2012 issue of Scientometrics, the paper indicates a significant increase of scholarly papers reporting that their study results supported the stated hypothesis, rather than disproving it:

This study analysed over 4,600 papers published in all disciplines between 1990 and 2007, measuring the frequency of papers that, having declared to have “tested” a hypothesis, reported a positive support for it. The overall frequency of positive supports has grown by over 22% between 1990 and 2007....

Fanelli notes some fascinating cultural differences in reporting negative findings, and included these wise words of warning:

A system that disfavours negative results not only distorts the scientific literature directly, but might also discourage high-risk projects and pressure scientists to fabricate and falsify their data.

Yes indeed.

[Hat tip @Mangan150]

See some prior posts on negative data in research: "Free the Dark Data in Failed Scientific Experiments," "Web site offers scientists access to lessons from failed experiments."

Free the "dark data" from scientific failures

I recently stumbled across "Freeing the Dark Data of Failed Scientific Experiments," by Thomas Goetz in Wired magazine. I had read it years ago, and its thesis still sounds as fresh today as it did back in 2007:

What happens to all the research that doesn't yield a dramatic outcome — or, worse, the opposite of what researchers had hoped? It ends up stuffed in some lab drawer. The result is a vast body of squandered knowledge that represents a waste of resources and a drag on scientific progress. This information — call it dark data — must be set free.

While the usefulness of negative data is being recognized, there are still powerful forces, organizational and human, working against freeing our dark data:

More and more, research is funded by commercial entities, which deem any results proprietary. And even among fair-minded academics, the pressures of time, tender, and tenure can make openness an afterthought. If their research is successful, many academics guard their data like Gollum, wringing all the publication opportunities they can out of it over years. If the research doesn't pan out, there's a strong incentive to move on, ASAP, and a disincentive to linger in eddies that may not advance one's job prospects.

One of the publications cited by Goetz is still going strong: "The Journal of Negative Results in BioMedicine," edited by Bjorn Olsen of Harvard Medical School, has possibly the most delicious description of any journal ever:

Journal of Negative Results in BioMedicine is an open access, peer-reviewed, online journal that promotes a discussion of unexpected, controversial, provocative and/or negative results in the context of current tenets.

When will all the results of scientific research be released into the wild? How can we free up this unused resource?

Release data from failed experiments! #2

Sam Loewenberg has a great opinion piece in the New York Times today that extols something we've advocated on this blog before: information from failed experiments must be shared rather than hidden.

His subject is an ambitious public-health project in Mumbai, India, run by the Municipal Corporation of Greater Mumbai and University College London. The project was not successful in driving meaningful infant-health improvement in the city despite trying a number of approaches. The researchers, to their credit, published their findings on the web, concluding, in part, "Facilitating urban community groups was feasible, and there was evidence of behaviour change, but we did not see population-level effects on health care or mortality."

Loewenberg's message:

What is noteworthy is that when the project did not work as planned, the team reported it openly and in detail, providing potentially valuable information for other researchers.

The risk is that too few people will follow. Especially in tough economic times, the pressure is on to show that they are getting bang for their buck. Last year an Obama administration official called on the aid community to adopt a “permanent campaign mind-set,” in which fund-raising and promotion are on the front burner. This creates an incentive to go for easy victories, highlight successes and bury failures. Even with the new fad in the aid world for metrics and impact assessments, their public reports are rarely forthcoming about missteps.

Here's a quotation we used in our previous post: "Science is very inefficient. You try an experiment, fail, try again, fail, try again, it works. And what works is what you publish. All the data about failure is wasted.”

Very true. Let's hope others follow the lead of the Mumbai government & UCL, and not reserve for publication only those projects that succeed.