Showing posts with label complexity. Show all posts
Showing posts with label complexity. Show all posts

Thursday, November 21, 2013

Mistake Bank Bookshelf: "The Logic of Failure"

This week’s selection on the bookshelf is an amazing book I discovered thanks to a tweet from Roxanne Persaud (@commutiny), The Logic Of Failure: Recognizing And Avoiding Error In Complex Situations by Dietrich Dörner. It was published in English in 1995, but is still (apart from a couple of dated references to the difficulty computers have playing world-class chess) amazingly current.

Dörner is professor emeritus of psychology at the University of Bamburg. In the book, he discusses many psychological simulations that illustrate how difficult it is for people to manage in complex environments. In the experiments, subjects are given a complex objective – say, to manage the well-being of an African tribal community by allocating water, seeds, etc. – and the ability to make periodic interventions. Due to the many psychological biases and blind spots we have, this task is very difficult, and very few participants can successfully keep things as good as they had been before the simulation began (many result in the collapse of the society). Participants develop tunnel vision, overcorrect for mistakes, and act before thinking. I have done some of these exercises and have suffered a similar fate. The few successful subjects observe before acting, develop an understanding of the interrelation of the system’s parts, and manage side effects. The average performance in this simulation might be cause to keep us humble about our ability to successfully intervene in the developing world.

One factor complicating the ability to manage complex systems is the time lag between cause and effect that these systems demonstrate. When a change is ordered, the result may not be seen for days/weeks/months, and may even be obscured by other factors. Dörner’s illustration of the difficulty subjects had with a relatively simple task containing such a lag (regulating the temperature in a room by adjusting a dial) will give pause to anyone thinking about proposed technological solutions to manage global warming.

There is a powerful amount of insight in this book. Some examples:

If, the moment something goes wrong, we no longer hold ourselves responsible but push the blame onto others, we guarantee that we remain ignorant of the real reasons for poor decisions, namely, inadequate plans and failure to anticipate the consequences….

This tendency to “oversteer” is characteristic of human interaction with dynamic systems. We let ourselves be guided not by development within the system, that is, by time differentials between sequential stages, but by the situation at each stage. We regulate the situation and not the process….

Clear goals will give us guidelines and criteria for assessing the appropriateness or inappropriateness of measures we might propose.

The Logic of Failure is a wise book with ample lessons. While it points out where our instincts get us into trouble, its point of view is generous and sympathetic. As a companion to Kahneman’s Thinking Fast or Slow or Taleb’s Antifragile, it is highly recommended.

Wednesday, October 9, 2013

Limitations of the "Swiss Cheese" analogy of complex failure

Kind readers of this site have pointed me to several papers related to complex-system failure. I'm grateful for their references - it's helped me process through a lot of my thinking on the subject. Please keep them coming!

One theme I've seen is that of the "Swiss Cheese analogy" for system failure. That is, individual activities or process steps are like slices of Swiss cheese - and the holes are individual errors. When the holes in a bunch of slices in a row line up, that is a systemic failure.

By this analogy, as long as we put actions in place to make sure the holes don't line up, we can avoid large failures. Those actions would be things like peer review, creating redundancy & backup systems, etc. This makes intuitive sense. Just prevent the holes from lining up! But there are two significant cases for which this analogy breaks down.

First is a system that is constantly evolving. The processes and practices that were put in place yesterday did not take into account the change that happened today. For example, a product sales plan is derailed by the increased adoption of a substitute product. Or a doctor's diagnosis is affected by having been involved in a minor car accident on the way to work.

In this example, new holes are being punched into the cheese. There are new failure states being created all the time, and reliance on process and behavior-type tools won't cover all the contingencies. Worse than that, a false confidence in these tools may allow people to overlook the brand new holes that have appeared.

The second situation is that of a highly unlikely but powerfully impactful event (a Black Swan). In this case, the probability of the event is so small that planners tend to overlook it (or at least underestimate its likelihood - what's the difference between 0.1% and 1%, anyway?). But when it occurs, the hole in the Swiss cheese is so huge that our safety measures can't cover it up.

For the second situation, there is a wealth of information in Nassim Taleb's book Antifragile and on his Facebook page.

The first one I'm thinking about right now. How do we create an environment where we both create processes to avoid holes lining up, and constantly keep everyone on the lookout for new holes?

Monday, October 7, 2013

4 pages of wisdom: "How Complex Systems Fail"

Roxanne Persaud (@commutiny) has shared an important paper in looking at mistakes and failure. It is "How Complex Systems Fail" (link - PDF) by Dr. Richard Cook, director of the Cognitive Technologies Laboratory at the University of Chicago. The 1998 paper is brief - only 4 pages long - and is in bullet form. In it, Dr. Cook summarizes his thoughts about complex human-developed systems - examples include power generation/distribution systems, health care, transportation.

Some of the headlines represent things I'm thinking about a lot these days: "Complex systems are intrinsically hazardous systems" - that is, failures will occur and not all can be avoided.

"Catastrophe requires multiple failures" - it occurs when "small, systemic failures join to create opportunity for an accident." This means that near misses are important and cannot be ignored - the right combination of near misses means disaster.

"Change introduces new forms of failure." Complex systems evolve, and what worked in the past may not work in the future due to the context and actors changing.

It's a terrific paper. One headline, "Human practitioners are the adaptable element of complex systems," makes me realize how important it is for workers in complex environments (that means all knowledge workers, managers, salespeople, customer service reps, nurses, etc.) to be aware they work in a complex environment. How many of those millions of workers don't even realize they are part of a system that breaks often, and that they can help repair by being cognizant of small mistakes and changes, and by sharing those lessons?

Thursday, October 3, 2013

Duke "Anatomy of an Error" course: is every mistake human error or a process defect?

I've been thinking about how to teach "mistake appreciation," for lack of a better term, to knowledge professionals. As such, I am very attuned to discussions of errors/mistakes/failures, including definitions and prescriptions. So I was very interested when @whatsthepont shared the Duke "Anatomy of an Error" online course (discussed earlier this week). Here's a graphic from my recent materials (adapted from Amy Edmondson's work):



Health care is a business that sits squarely in the complex domain - the center of the spectrum. And as you can see, failures are unavoidable. How important, therefore, is it to teach people involved in this process how to understand, prevent and learn from errors? Very important, of course. So the Duke course is admirable.

However, my concern about the course is that it seems to hold a point of view that if workers acted carefully enough, and process designers were thorough enough, that errors in health care could be eliminated completely. Here's an example of what I'm talking about from the course:

Because humans are fallible, we must rely upon systems and back-ups to prevent or detect errors before they can cause harm. Unfortunately, our systems are not always designed well to achieve this. System and design factors that can lead to bad outcomes include:
Complexity:
  • Too many steps
  • Too many people (communication issues)
Workload:
  • Too heavy or too light (performance is best when workload is moderate)
  • Too much reliance on human vigilance/monitoring
Poor design:
  • Focus on functionality, while ignoring the real-life user 

This presumes that dealing with all the bulleted items will avoid all possible bad outcomes. This thinking is just wrong. Inherent in a complex process is changing interactions between different people and the environment. Things evolve. Health care is no different. As a result, unexpected results can occur even if workers work with the utmost care and the processes have been designed to the utmost quality. Because things change around them.

This is more reason to take a more open view of error than that simply created by tired workers or inadequate processes. And to thrive with this open view requires a culture that does not work by "stored patterns of pre-programmed instructions," but that is mindful, aware, and welcomes observations and insights from anyone in the organization (not just doctors).

In addition to "Anatomy of an Error," Duke Medical Center would be well advised to teach their workers to be very aware of surprises and disappointments - to use a spy novel term, "expect the unexpected." If well-trained, well-rested staff perform a well-designed process and something still goes awry, that is not a defect, it is vital information. It means something has not been taken into account, that the environment has changed, and a cool, clearheaded review is in order. It could be the appearance of a new strain of resistant bacteria, a new designer drug, or simply a unique patient situation. The worst possible step is to hide the error or to engage in "quiet fixing."

This is the next level of training that groups like Duke need to embark on. They should be commended for training their staff on this weird topic of errors and mistakes. But they can't stop there; there's much left to do.

Friday, September 27, 2013

Mistake Bank Bookshelf: Tim Harford’s “Adapt”

Tim has been a frequent inspiration to this site for the past few years, initially for his TED talk entitled “Trial, Error and the God Complex.” His 2011 book Adapt: Why Success Always Starts with Failure is this week’s selection.

His background is in economics; he writes the “Undercover Economist” column for the Financial Times, and in Adapt the 2008 financial crisis is an exception of the failure-begetting-success theme that he strikes throughout the book. He also takes up the on-the-ground action of the wars in Iraq and Afghanistan, the changing composition of the Fortune 500, and the invention of the toaster as examples of adaptation all around us.

The lens used to investigate these events is that of complex adaptive systems – the methods used by biological evolution. It’s a fascinating book and an important addition to the Bookshelf.

Here’s a section describing a famous project by the psychologist Philip Tetlock:

[Tetlock] rounded up nearly three hundred experts – by which he meant people whose job it was to comment or advise on political and economic trends. They were a formidable bunch: political scientists, economists, lawyers and diplomats. There were spooks and think-tankers, journalists and academics. Over half of them had PhDs; almost all had postgraduate degrees. And Tetlock’s method for evaluating the quality of their expert judgment was to pin the experts down: he asked them to make specific, quantifiable forecasts – answering 27,450 of his questions between them – and then waited to see whether their forecasts came true. They rarely did. The experts failed, and their failure to forecast the future is a symptom of their failure to understand fully the complexities of the present.

[Tim has a new book out called The Undercover Economist Strikes Back.]

Tuesday, February 1, 2011

Scott Berkun on learning from mistakes

While working on The Mistake Bank, I found this essay by Scott Berkun, author of books like "The Myths of Innovation." Upon reading it, I was struck by how well-articulated his arguments are and how closely his thinking relates to what we're trying to do with The Mistake Bank.

It starts off like this:

You can only learn from a mistake after you admit you’ve made it. As soon as you start blaming other people (or the universe itself) you distance yourself from any possible lesson. But if you courageously stand up and honestly say “This is my mistake and I am responsible” the possibilities for learning will move towards you. Admission of a mistake, even if only privately to yourself, makes learning possible by moving the focus away from blame assignment and towards understanding. Wise people admit their mistakes easily. They know progress accelerates when they do.

And there's lots more. The section entitled "How to Handle Complex Mistakes" is particularly relevant--as is his discussion on the importance of keeping a sense of humor about yourself. Please give Scott's essay a read.