Failure guru Amy Edmondson deconstructs the Healthcare.gov fiasco

Amid all the breathless news coverage of the failed rollout of the Obamacare Healthcare.gov website, we now have some genuine analysis, courtesy of one of my heroes, Amy Edmondson of Harvard Business School ("The Mistakes Behind Healthcare.gov Are Probably Lurking In Your Company, Too"). She may be more qualified than anyone to weigh in, given her deep research experience in learning from mistakes and failure in very complex situations (including healthcare). A couple of potent excerpts:

Healthcare.gov is a good example of the importance of learning small and fast, rather than rolling out a risky new product or service launch all at once. Cycling out in phases includes the expectation of early failures – and demands all hands on deck to learn from them along the way. A roll-out, in contrast, implies that something is all set, ready to go — like a carpet. All it needs is a bit of momentum to propel it forward. For complex initiatives, of course, this is simply not the case. Getting people motivated enough to change is not the real challenge; it’s getting them engaged enough to learn — to become part of a discovery process.

and...

Managers must make it clear that they understand that excellent performance does not mean not making mistakes — it means learning quickly from mistakes and sharing the lessons widely.

Best Books of the Year 2012

These are the best books I've read this year concerning human error, making mistakes and learning from them. (They may have been published in prior years.)

1. "Teaming: How Organizations Learn, Innovate, and Compete in the Knowledge Economy," Amy Edmondson. The preeminent researcher on organizational learning and development sums up her 20-year career in this book. It covers teamwork, collaboration and leadership, and is more full of common sense than a shelfload of self-help books. Money quote: "Facing the potential for larger failure when smaller failures interact, leaders in complex organizations must promote resiliency by acknowledging that failure is inevitable, making it psychologically safe to report and discuss problems, and promoting habits of vigilance that support rapid detection and responsiveness."

2. "The Signal and the Noise: Why Most Predictions Fail but Some Don't," Nate Silver. Almost a companion piece to Kahneman's "Thinking, Fast and Slow," which was on last year's list. Silver attacks misplaced self-confidence in human forecasting and shows us that by embracing uncertainty and weighing the odds, we are capable of making far better predictions than we usually do.

3. "From Lemons to Lemonade: Squeeze Every Last Drop of Success Out of Your Mistakes," Dean Shepherd. A researcher from Indiana University takes a personal-psychology viewpoint on the mistakes question. With very useful discussions on various ways to bounce back from mistakes, and why we feel so bad when we fail, this book has a prominent place in my "Mistake Learner's Bookshelf." Money quote: "Failure is not the opposite of success."

4. "Adapt: Why Success Always Starts with Failure," Tim Harford. Looking at human organizations as complex-adaptive systems, Harford shows many examples of groups finding success by responding to initial failure by adapting and learning; as well as an equal number of those who didn't adapt and failed utterly as a result. His views, coming from an economics background, and Amy Edmondson's, from an organizational development perspective, mesh nicely.

5. "Anna Karenina," by Leo Tolstoy. Translated by Richard Pevear and Larissa Volokhonsky. If you want to learn about human fallibility and the trouble it can cause, you can't do better than to consult the Russian masters. If "War and Peace" illustrated bureaucratic and organizational folly, this book does the same with social relations. And, did you hear there's a movie out?

"Teaming" Day 5: Final Thoughts

This is the last in our weeklong series of posts on "Teaming: How Organizations Learn, Innovate, and Compete in the Knowledge Economy," by Amy Edmondson.

I will wrap up our week by highlighting some of the best quotes in "Teaming." I've added a couple of editorial comments in [square brackets]:

When facing an uncertain path forward, trying something that fails, then figuring our what works instead, is the very essence of good performance. Great performance, however, is trying something that fails, figuring out what works instead, and telling your colleagues all about it - about both the success and the failure. (pp 29-30)

Despite rhetoric to the contrary, many of us still expect ourselves and others to get things right the first time. We view failures as unacceptable. We give directives to those below, and look for direction from supervisors above. (p 40)

When people frame a task as a "performance situation" they are more risk averse and less willing to persist through obstacles than when the same task is framed as a "learning situation." Not only do people adopting a learning frame persist longer in unfamiliar, challenging tasks, but they ultimately learn more as a result. In addition, people with a performance frame engage in less experimentation and innovation and are less likely to formulate new strategies in difficult situations. Instead they're more likely to fall back on ineffective strategies they have used previously. (p 86)

To learn from mistakes and missteps, organizations must employ new and better ways to go beyond lessons that are superficial (procedures weren't followed) or self-serving (the market just wasn't ready for our great new product). This requires jettisoning old cultural beliefs and stereotypical notions of success and replacing them with a new paradigm that recognizes that some failures are inevitable in today's complex work organizations and that successful organizations will be those that catch, correct, and learn from failures quickly. (p 150)

People tend to be more comfortable considering evidence that supports what they believe, denying responsibility for failures, and attributing problems to others. (p 155)

The difference between failures that are truly blameworthy and those that are simply treated as blameworthy reveals a gap between logic and practice. [I would add that it reveals a gap between a person's vision, or what she hopes to be true, and raw reality.] (p 160)

Engineers' or scientists' intuition can be telling them for weeks that a project has fatal flaws, but making the formal decision to call it a failure may be delayed for months. Considerable resources are saved when such projects are stopped in a timely way and people are freed up to explore the next potential innovation. [I've experienced this in business-to-business sales environments. The declaration of a loss can take weeks or months, chewing up resources all the while.] (p 174)

Most of us would prefer to have reliable solutions to the problems we face, and we certainly like to feel that we are good at what we do. But execution-as-learning requires us to accept our individual and collective fallibility. (p 225)

You can find all our posts related to "Teaming" here.

"Teaming" Day 4: Varying reasons for failure

This is the fourth in our weeklong series of posts on "Teaming: How Organizations Learn, Innovate, and Compete in the Knowledge Economy," by Amy Edmondson.

Blame and fault-finding is as old as humanity. We learn this early in life: "It wasn't my fault. They did it!" For mistakes in the workplace, the context of the situation matters a great deal when deciding whether an unexpected result is, in Amy Edmondson's words, "praiseworthy" or "blameworthy." She describes a spectrum of situations that explain failures, spanning from ... to ... Here's the list of reasons for failure, from most blameworthy to highly praiseworthy. Items in quotes are taken from Edmondson's description.

Deviance - failure due to not following prescribed procedures is a cause for reprimand, discipline, or firing.

Inattention - "inadvertent deviation" from specifications. This sometimes happens to me when I forget to attend a conference call I'd committed to. Blameworthy!

Lack of Ability - failure due to lack of skills, training, etc. Blame here rests partially on the worker, and partially on management, who need to assess and verify ability on an ongoing basis.

Task Challenge - failure due to the inherent difficulty of a task. Failing to get a hit in baseball seven out of ten times is simply a facet of the game.

Process Complexity - a breakdown in a system due to "novel interactions." Not blameworthy, in fact workers need to be encouraged to report these types of failures, so the novel interactions can be diagnosed and training developed to overcome the new obstacle.

Uncertainty - failure due to circumstances unforeseen at the time an action was taken. This type of failure happens often when there is a long lag between action and outcome. For example, locking in the price of next year's natural gas supply, and then seeing prices fall dramatically. Failures due to uncertainty are reason to evaluate how decisions are made and whether steps can be taken to manage uncertainty or its consequences. (People who succeed in an uncertain environment get undue credit. This discussion by Daniel Kahneman on stock-pickers addresses this point.)

Hypothesis Testing - "an experiment conducted to prove that an idea or a design will succeed or fail." Whether the hypothesis is proven or disproven, you've gained valuable information as a basis for higher-stakes decisions.

Exploratory Testing - an attempt to probe and understand a novel area. Many of these attempts will fail, but even a small percentage of successes will yield large rewards. Praiseworthy!

This quote was cited in an earlier post, but it's worth repeating when talking about how to assess whether someone should be blamed or praised for a failure:

When I ask executives to...estimate what percentage of failures in their organizations are caused by blameworthy events, the answers usually come back between 2 and 5 percent. But when I then ask what percentage of failures are treated as if caused by blameworthy events, after a pause or laugh, their responses often yield a much higher number in the 70-90 percent range.

You can find all our posts related to "Teaming" here.

"Teaming" Day 3: Three Types of Failure

This is our third post on Amy Edmondson's "Teaming: How Organizations Learn, Innovate, and Compete in the Knowledge Economy."

Chapter 5 is called "Failing Better to Succeed Faster" and it's full of wonderful insights about learning from mistakes. Amy defines three types of failure so well and so crisply that I won't paraphrase the definitions, but will quote them directly:

• Preventable Failures: process deviations in well-understood domains, usually caused by behavior, skill or support deficits. 

• Complex Failures: process or system breakdowns that arise due to inherent uncertainty and may or may not be identified in time to prevent consequential accidents.

• Intelligent Failures: the unsuccessful trials that occur as part of thoughtful experiments and provide valuable new information or data.

Of these three, complex may be the most difficult to manage. In routine environments, preventable failures should be easy to spot and to diagnose. In innovative environments, tolerance for failure is built in. In complex environments, however, deviation from expectations could be a "near miss" that could lead to a "consequential accident." It could be valuable information about a change in the environment. It might be an anomaly, due to events out of your control. How do you learn and improve in complex operations?

When reviewing these types of failures, first, don't blame individuals. Second, don't jump to the first explanation that springs to mind. [Quote from "Teaming": "The primary danger in failure analysis is that people tend to leap prematurely to conclusions, unless the analysis emphasizes a careful consideration of all possible causes and effects."] Finally, take ownership of the failure by looking at things you and your team can do differently when confronted with a similar situation in the future.

Here's a real example. I was involved on a sales campaign with a large customer. We crafted a very innovative solution for them. The team at the customer was very confident in our capabilities and had justified this project. We had negotiated price, and contractual points. Finally, only one more signature was needed by a customer executive.

He did not sign.

We who had worked on the opportunity were crestfallen. We had done everything we could to win this deal, and had nothing to show for it. Our first reaction was to chalk the failure up to bad luck, or fate, or even an evil, conniving customer executive plotting to screw us.

But as we reflected on the situation, it became clear that we had not done everything we could have. We had not fully understood who at the customer could make a commitment for this business. And we hadn't gotten in contact with that person to see if this project helped him with his objectives, or whether there were other projects he felt more passionate about. So, the learning was this: the next time we engaged on a sales campaign, we would, as early as possible, understand who would sign the contract and assess how to enlist that person's support. And if we couldn't enlist that support, consider stopping the pursuit.

You can find all our posts related to "Teaming" here.

"Teaming" Day 2 - The Process Knowledge Spectrum

This is another post on "Teaming: How Organizations Learn, Innovate, and Compete in the Knowledge Economy," by Amy Edmondson.

One concept in the book is the "Process Knowledge Spectrum." This spectrum describes three broad types of business operations that require different approaches to learning and tolerances for mistakes. It runs from Routine Operations through Complex Operations and to Innovative Operations on the other pole.

Routine Operations are business processes that are well-defined and for which conformance to specification is crucial. A McDonald's hamburger is made the same way worldwide. Mistakes in this environment are unexpected and due to inability or unwillingness to following procedure. Learning in this environment involves observing and noting small changes that can improve outcomes. The classic learning environment for Routine Operations is the Toyota Production System.

Complex Operations happen over and over again, but never exactly the same way twice. For example, customer service interactions or sales engagements. The people involved are different, the context has changed, and the environment has evolved. So the idea of "best practice" as used in routine operations is not applicable. (This hasn't stopped many organizations from trying unsuccessfully to make it so, but that's a subject for another post.) Mistakes in this complex operations often indicate, as Amy points out, system breakdowns. They can also indicate weak signals of change in the environment. They are rarely the fault of an individual, and treating them as such has a very negative side effect - it promotes hiding of mistakes, and, therefore, hiding the system breakdown or weak signal that is vital for learning and improvement.

Examples of Innovative Operations are development of a brand-new product or starting up a new business venture. In these operations, much is unknown, and only by developing hypotheses, probing and experimenting can learning occur. Innovators are often unperturbed by mistakes, and in fact welcome them. (Consider this quote from a famous innovator, Thomas Edison: "Negative results are just what I want. They’re just as valuable to me as positive results. I can never find the thing that does the job best until I find the ones that don’t.")

Uncertainty increases along the Process Knowledge Spectrum, from low uncertainty in Routine Operations to very high uncertainty in Innovative Operations. That amount of uncertainty indicates the usefulness of standard practice and the predictability of outcomes. In innovative projects, outcomes are highly unpredictable, experimentation is important and tolerance for failure needs to be high.

This spectrum is very useful in analyzing mistakes. If you are involved in a project in which the result is not what you expect, think about which of the above definitions fits the project best. Is it routine, complex or innovative? Knowing this will help you understand how to deal with the mistake and how to manage for mistakes going forward.

You can find all our posts related to "Teaming" here.

Amy Edmondson's "Teaming"

I'm so delighted that Amy Edmondson has published "Teaming: How Organizations Learn, Innovate, and Compete in the Knowledge Economy," a summation of her 20-year study of organizational learning & performance. Amy's ideas provide one of the pillars for this site and have helped shape my thoughts about perfectionism, leadership and teamwork.

I'm equally delighted that the book is utterly worthy of the expectations I had for it. Impeccably organized and crisply written, it sets out Amy's arguments so cleanly that you forget how radical her ideas are. At least, until you read something like this:

When I ask executives to...estimate what percentage of failures in their organizations are caused by blameworthy events, the answers usually come back between 2 and 5 percent. But when I then ask what percentage of failures are treated as if caused by blameworthy events, after a pause or laugh, their responses often yield a much higher number in the 70-90 percent range.

If you are looking for an introduction to the ideas we're exploring on The Mistake Bank, please read Chapter 5, "Failing Better to Learn Faster," several times. And then all the other chapters.

We'll spend the rest of the week sharing a few of the countless valuable nuggets from "Teaming." Expect it to be on our year-end "best of" list.

A full collection of posts related to Amy Edmondson's work can be found here.

Novartis' Joseph Jimenez: forecasting errors caused by culture, not process

This story, from Novartis CEO Joseph Jimenez via Adam Bryant's Corner Office series in the NY Times, has echoes of Amy Edmondson's research into high-performing hospital teams. Read the story, and I'll tell you what I think the connection is:

Q. What are the most important leadership lessons you’ve learned?

A. One occurred when I was a division president of another company. I was sent in to turn the division around after four years of underperformance. It was a declining business. And when I got there, I completely misdiagnosed the problem. I said: “Look. We’re missing our forecast every month. What’s wrong?” I brought in a consulting firm, and we looked at what was wrong. And the answer was that we had a bad sales and operations planning process, where salespeople, marketing people and operations people were supposed to come together and plan out the next 18 months and then forecast off of that. So I said: “O.K. We’re going to fix this. We’re going to have the consulting team come in and help us make that a better, more robust process, with more analytics.”

And it turned out it wasn’t at all about analytics. Because once we did that, and we put that new process in place, we still continued to miss forecasts. So I thought, “Something’s really wrong here.” I brought in a behavioral psychologist, and I said: “Look, either I’m misdiagnosing the problem or something’s fundamentally wrong in this organization. Come and help me figure it out.” She came in with her team and about four weeks later came back and said: “This isn’t about skills or about process. You have a fundamental behavioral issue in the organization. People aren’t telling the truth. So at all levels of the organization, they’ll come together, and they’ll say, ‘Here’s our forecast for the month.’ And they won’t believe it. They know they’re not going to hit it when they’re saying it.” The thing she taught me — and this sounds obvious — is that behavior is a function of consequence. We had to change the behavior in the organization so that people felt safe to bring bad news. And I looked in the mirror, and I realized I was part of the problem. I didn’t want to hear the bad news, either. So I had to change how I behaved, and start to thank people for bringing me bad news.

Edmondson found that hospital teams with the highest outcomes also had the highest reported incidence of mistakes. When she dug into this paradox, she found that the high-performing teams had the psychological safety to share mistakes so they could be fixed and avoided in the future. The poorer-performing teams tended to cover up their mistakes.

Amy Edmondson - in today's complex organizations, failures are inevitable

Amy Edmondson is an old friend of this site. Here she is featured in a video from the series from Berghs College of Communication's 2011 Exhibition.

Amy C. Edmondson – on the fear of failure. from Berghs' Exhibition '11 on Vimeo.

More information on the Berghs Exhibition here.

Amy Edmondson on learning from failure

Amy Edmondson, Novartis Professor of Leadership and Management at Harvard Business School, discusses her research into learning from failure. This video was posted as part of the Harvard Business Review "Failure Issue."



Highlights:

2:48 - The skills of learning from failure
3:50 - Lessons from the Columbia shuttle tragedy
8:40 - Getting workers to speak up about errors and concerns
11:10 - Raising issues with the intent to make things better

Amy Edmondson on mistakes in the workplace

In the January 19, 2009 issue, the New York Times business section covered one of our favorite subjects, as part of its "Career Couch" series: "Making the Most of Your Workplace Mistakes." Even better, they interviewed an important researcher on this subject: Amy Edmondson of Harvard Business School (discussed many times on my blog, including here and here).

Here's my favorite quote from the article:

In any job that requires continuing thought and judgment, we need to be “aware of the huge potential of things to go wrong,” Professor Edmondson said, because “we all can handle only so much cognitively at a time.”

"Failure is just evidence that you haven't mastered the task yet"

Harvard Business School professor Amy Edmondson has an important article in the July 2008 HBR that says a lot of things about learning in the business world. The following excerpt has some particular relevancy to The Mistake Bank:

In her research on individual mind-set differences, Stanford psychologist Carol Dweck has shown that the way children view a task affects their persistence and performance over time. Some children think of human ability or intelligence as fixed and, consequently, think of school tasks as performance opportunities--moments of truth that prove whether they're smart [JC note: call it the Hermione Granger syndrome]. For these children, performing poorly on an assignment or a test would demonstrate that they lacked intelligence rather than indicating that they had more to learn. Believing that the point of execution is to demonstrate competence, they go out of their way to pick easier tasks. Of course, this means they lose out when it comes to learning. This same mind-set encourages managers to admire and expect to be rewarded for decisiveness, efficiency, and action rather than for reflection, inquiry, and collaboration, the uncertainty of which makes them uncomfortable. Like the children who have learned to shun new challenges, these managers avoid, and help others avoid, the risks of questions and experiments.

In psychologically safe environments, people are willing to offer up ideas, questions, concerns - they are even willing to fail - and when they do, they learn. In her studies, Dweck found that some children - those who early on were rewarded for effort and creativity more than for simply giving the right answer - see intelligence as something malleable that improves with attention and effort. Tasks are opportunities for learning; failure is just evidence that they haven't mastered the task yet. Driven by curiosity about what will and will not work, they experiment. When things don't pan out, the don't give up or see themselves as inadequate. They pay attention to what went wrong and try something different next time. In adults, such a mind-set allows managers to strike the right tone of openness, humility, curiosity, and humor in ways that encourage their teams to learn."

Dweck's research on "growth" vs. "fixed" mindsets has a lot to say about whether we can learn from our mistakes or be paralyzed by them.

Great innovation requires great teams, candor, and acceptance of mistakes

Some research by Harvard Business School professor Amy Edmondson on team learning is important to our discussions of sharing and learning from mistakes. The research centered on explaining a paradox--why in her studies did excellent teams make more errors than poor teams?

The answer, as you might expect, was greater candor and its corollary, greater confidence and openness to learning. Good teams simply communicated better, and, in a learning environment, that meant surfacing and talking about mistakes.

In a discussion about the topic with HBS Working Knowledge, professor Edmondson summarized her findings thusly:

In well-led teams, a climate of openness could make it easier to report and discuss errors—compared to teams with poor relationships or with punitive leaders. The good teams, according to this interpretation, don't make more mistakes, they report more. When I suggested this to physicians involved in the study, they were skeptical. Their response was understandable: With a research grant for the purpose of identifying the error rate, this idea was decidedly unwelcome. My interpretation of the data suggested that we might not be finding the definitive error rate—and further errors might be systematically underreported in certain units but not others. Their skepticism forced me to work hard to develop ways to support my proposition, which ultimately they came to see as reasonable, if not obvious in retrospect.

Once again, we see that learning in adults means supressing instincts for self-protection, defying organizational incentives to conform and be "team players," and ignoring ingrained concepts like division of labor and roles/responsibilities.

This is from a 2006 working paper on the subject, "When Learning and Performance Are At Odds" from Professor Edmondson and her collaborator, Sara Singer:

...Effectively conducting an analysis of a failure requires a spirit of inquiry and openness, patience, and a tolerance for ambiguity. Such an inquiry orientation is characterized by the perception among group members that multiple alternatives exist, frequent dissent, deepening understanding of issues and development of new possibilities, filling gaps in knowledge through combining information sources, and awareness of each others’ reasoning and its implications(Argyris et al., 1978). Such an orientation can counteract common group process failures. Learning about the perspectives, ideas, experiences, and concerns of others when facing uncertainty and high stakes decisions, is critical to making appropriate choices.

Looking at this through the prism of innovation, you can see how using the whole disorderly team, how arguing and soliciting dissenting views is essential. Innovation means confronting the unknown, the complex, the ill-defined. Mistakes are to be expected, not avoided. Confronting, embracing failure, then gathering the entire teams's viewpoints on what didn't work and how to fix it, then stepping back and trying a different tack, is essential. Locating dead ends and understanding failure quickly and changing course leads to faster innovation development, lower cost and higher probability of eventual success.