Web site offers scientists access to lessons from failed experiments

The New York Times Bits blog profiled the website ResearchGate, which gathers lessons from failed scientific experiments and makes them available to other researchers:

“Science is very inefficient,” says Ijad Madisch, founder of a Web site called ResearchGate. “You try an experiment, fail, try again, fail, try again, it works. And what works is what you publish. All the data about failure is wasted.”

Begun in 2008, ResearchGate claims to have 11,000 research and educational institutions among its users. It aims to be a place where people can share what they learned in the failed experiments. Some of this is documented, but a great deal more takes place across chat rooms where scientists informally exchange information.

The ResearchGate site also includes a user post asking for a database of data from failed experiments, a proposal I've seen before (but can't recall where). It's a great idea:

I think developing a database for the storage of failed experiments makes sense for experiments that lost the chance to be shown in published articles. Everyone could upload their experimental results which they find unsatisfactory, including the experiment method, experiment data, and the experiment purpose.

This experimental data can give a good experience and lessons to the researchers who want to perform similar experiments. They can improve their experiment based on the failing example. Someone might also give a different opinion for the unreasonable experimental data, and then test their idea. Maybe a novel theory can be developed from these failed experiments. Or, like the Post It, the unsuccessful experiment data could be used to solve other problems.

Vijay Govindarajan on how to learn through failing cheaply & quickly

This brief video from Vijay Govindarajan of the Tuck School at Dartmouth University illustrates a simple method for creating an effective innovation experiment. His recommendation: "Test your most critical unknowns as early and as inexpensively as possible."

This has some resonance with the "Smart Mistakes" article I recently posted on The 99 Percent.



Govindarajan is the co-author of "Reverse Innovation: Create Far From Home, Win Everywhere." We posted on that book earlier.

From "Reverse Innovation": PepsiCo India's experience of trial and error

The new book "Reverse Innovation: Create Far From Home, Win Everywhere" by Vijay Govindarajan and Chris Trimble, describes the phenomenon of new products being born in emerging markets, with radically different value propositions and price points, and spreading to developed markets. This approach is the reverse of the traditional model for Western companies, which create (expensive) products at home, and then adapt them to developing markets.

This excerpt describes some of PepsiCo India's experience developing a new snack cracker named Aliva. It beautifully highlights the inevitability of mistakes when you are trying something new, and that persistently pushing through and learning from those mistakes are hallmarks of a successful product launch.

From its inception to its 2009 launch, the Aliva project took nearly four years. Aliva was evaluated against criteria that took full account of potential uncertainties. Such latitude was indispensable. Aliva had to make its way through a predictably fraught gestation. There were plenty of bumps in the road, and plenty to learn on the way.

Aliva's most vexing challenge was its packaging. Packages are hugely important to snack food performance. If snacks had remained in the era of the general-store cracker barrel, great branding opportunities would never have materialized. Aliva's packaging needed to be as distinctive as the shape of the cracker. The packaging had to communicate that Aliva was both healthy and fun. Decisions about the package would have implications for Aliva's texture and shape, the way the cracker was produced (through baking), and the attractiveness of the offering at the point of sale.

The Aliva bag featured a number of innovations. [Program manager Vidur] Vyas claims that nothing like it had ever been tried before. It was to be made from new materials on brand-new - and untested - machinery. The bag was designed to be flat on the bottom. Unlike typical snack bags in the United States, it could stand up straight on a retail shelf, tabletop, or counter. The packaging material was therefore heavier and stiffer than conventional plastic film. It turned out that a more rugged package could actually be made using only two laminate layers, not three. This solution was both more cost-effective and environmentally friendly.

The package specifications needed to address certain constraints of local infrastructure. It often took a long time to distribute perishable goods through a vast, predominantly rural retail network. Crackers can spoil more quickly than other types of snacks. Aliva therefore had to be protected from spoilage as well as breakage. A rugged, lightproof, hermetic package was key.

Vyas and his team endured a perfect storm of complications on the way to satisfying these needs. Because the Aliva bag was a first-of-its-kind package design, it seemed that every element of the package's structure and manufacture either had to be invented or endlessly troubleshot. To start with, the new packaging machinery was touchy. In limited test runs, things seemed fine. But once Aliva launched, in May 2009, problems cropped up during production-scale runs, particularly with the heat seal at the top of the bag. So, new material had to be designed. This required help from squadrons of global experts on polymers and lamination technologies.

There were nettlesome challenges on other fronts as well. Because Aliva would rely on a new baking system, which had only recently been used for the first time to produce cookies in PepsiCo's Mexico region, Vyas's team needed time and technical guidance to learn how to operate it reliably.

Finally, the team aspired to create a cracker in an eccentric triangular shape. The cracker's unique design was considered an important aspect of the values the brand would communicate. The triangle shape was meant to connote speed, stimulation, and taste. The triangle's curved edges were meant to connote health. At first, however, the crackers suffered unacceptable levels of breakage. Coming up with a workable version - a cracker with a low rate of breakage and a pleasing combination of textures - required innumerable trials.

But if Aliva's journey to market had an unusual share of difficulties, that is only because it was forging entirely new paths in a number of areas. To its credit, PepsiCo patiently tolerated a high degree of necessary experimentation... with packaging, with the baking system, and with the architecture of the cracker itself. (pp 167-168)

From "Reverse Innovation: Create Far From Home, Win Everywhere," by Vijay Govindarajan and Chris Trimble. Published by Harvard Business Review Press. Excerpted by permission.

Thomas Edison quote of the day

"Negative results are just what I want. They’re just as valuable to me as positive results. I can never find the thing that does the job best until I find the ones that don’t."

via Goodreads.

More on Edison's life here.

[Photo: part of Edison's lab complex in West Orange, NJ]

From FastCo Design: Only Certain Failures Help You Innovate

This is a neat post from FastCo Design, written by Jamer Hunt, organizing types of failures and which are productive or not productive ("Among Six Types of Failures, Only a Few Help You Innovate"). Here are his types; he organizes them from most severe to least:

• Abject Failure (aka a disaster)
• Structural Failure (requires significant rework and recovery)
• Glorious Failure (not completely sure what this is; he references the Jamaican bobsled team; I might call it a trivial failure. Something blows up, but no one is hurt and little damage done)
• Common Failure ("what the apology was invented for")
• Version Failure (something that doesn't work great but is the basis for improvement, such as Microsoft anything 1.0)
• Predicted Failure (the result of experimentation and prototyping)

Of these, only the last two are helpful for innovation in Hunt's view.

There's a lot to think about here. As Hunt writes, "we may now be able to recognize that there are valuable kinds of failure that are essential to innovation processes (version and predicted), while acknowledging that there are other types of failures that do little good."

I think what Hunt has done is very useful as a starting point for dialogue. The Mistake Bank is set up to share failures and learn from them. The diversity of stories and viewpoints here shows that there are many ways to view failures - and that many more types are productive than may be seen at first glance. We need to leave room for "the Happy Accident."

Benefiting from "deliberate" mistakes

In the June 2006 Harvard Business Review, Paul Schoemaker and Robert Gunther write about ways companies get bound up in their own assumptions, and thereby miss important opportunities for growth or improvement.

Their proposal? Deliberately make a "mistake" by doing something that violates an assumption you hold, to test whether the assumption needs to be altered. (Their article can be found here. Note: you need to be a subscriber to access the full contents online.)

Schoemaker and Gunther cite an example where the Bell System decided to forgo security deposits from some customers their systems had identified as credit risks. This was done in a controlled way, with a small but significant sample size, in order to test their approach to dealing with credit-risky customers. They found that their rules for requiring deposits were too strict, and that many of the customers who otherwise would have not opened an account (because they couldn't afford the up-front deposit) turned out to be reliable payers. Adjusting the processes based on the test added, according to the article, $137 million per year to the Bell System's profits.

Here are some highlights from the article:

Although organizations need to make mistakes in order to improve, they go to great lengths to avoid anything resembling an error. That’s because most companies are designed for optimum performance rather than learning, and mistakes are seen as defects that need to be minimized. Executives, moreover, perceive that flawless execution is what makes them valuable to the organization. In business (with the possible exception of venture capital firms and entrepreneurial start-ups), an executive’s reputation and rewards are typically based on the height of his or her successes, not on the depth of learning from failures.

and

Many managers recognize the value of experimentation, but they usually design experiments to confirm their initial assumptions. An advertising company typically may try different approaches to see which tactics work best but won’t run an ad that it presumes will fail. Experiments of this type aren’t deliberate mistakes. True deliberate mistakes are expected, on the basis of current assumptions, to fail and not be worth the cost of the experiment. According to conventional wisdom, they have a negative expected value. But if such a mistake unexpectedly succeeds, then it has undermined at least one current assumption (and, often, more). That is what creates opportunities for profitable learning.

Have you upended any of your assumptions recently? Perhaps it's time you made a few more mistakes--on purpose.

Marissa Mayer on Google’s Biggest Mistakes

It's always refreshing to hear mistake stories from companies whose successes have given them mythical status. Consider Google. There have been so many stories written about their engineering prowess (here), their unique culture (here & here), etc., that it's easy to view them as mistakes (Though it's fair to say their public image has taken a bit of a beating recently.)

Anyway, here's Google exec Marissa Mayer on some of Google's mistakes along the way (she also does a nice job of discussing the value of making mistakes in an innovative business):



(Hat tip The Next Web)

Fran Ten of West Indian Girl - not using the "setup we know works" on a radio performance

This story is from Fran Ten, who is the bassist for the LA band West Indian Girl. Fran shared a story about a radio appearance, involving a suggestion his West Indian Girl bandmate, Rob James, had about using a new instrument.

There was one time, we had a radio gig, and Rob thought that someone should play a harmonium, you know, that Indian instrument? On his acoustic set. And it sounded like shit. Right? It sounded horrible.

I said, “You know, Rob, that was a mistake. We are never bringing a harmonium again to an acoustic radio show. You’re just going with your guitar, or this setup we know that works.”

But we tried it. At least we tried it. Business-wise, you have to keep making mistakes. Isn’t that how you grow?

The Mistake Bank Manifesto

I've been reading the new book "Senior Leadership Teams: How to Make Them Great," by Ruth Wageman, Debra Nunes, James Burruss and Richard Hackman. Very close to the end of the book I found a passage that is a better explanation of what's behind the Mistake Bank than anything I could write myself. While it's focused on senior leaders, I think the ideas work for anyone who has a job or owns a business. [I'll do a full review of the book next week. Sneak preview: it's very good.]

To learn continuously... requires that senior leaders move beyond well-practiced leadership habits and well-learned personal models of what makes for a great leadership team. What's needed is active experimentation with new and unfamiliar leadership strategies, and whenever there is experimentation expect that there will also be failure.. More often than not, trying out a new grip or swing in golf or tennis results in worsened performance for a while. But these experiments also generate learnings that cannot be had otherwise. The same is true for experimentation with leadership strategies and skills.

In fact, error and failure always provide more opportunities for learning than do success and achievement, because failures generate data that you can mine for insight into how you might improve your assumptions or your mental model of team leadership. Indeed, the bigger the failure, the greater the learning opportunity. To learn from failure requires that you ask questions that arouse anxiety (for example, about the validity of your deeply-held assumptions or about personal flaws in your diagnosis or execution abilities). Learning from failure also requires that you gather data that can help answer those questions and then adapt your mental models and your behavior. These activities are not natural or comfortable acts, and they are especially unnatural for successful people who have limited experience in learning how to learn from error and failure. (p. 204)

Copyright 2008 Harvard Business Press