A good model can be useful even when it fails

As the rains from Hurricane Sandy fall around us here in Central Pennsylvania, I'm checking out various computer models about the track of the storm, and am reminded about this quote from Nate Silver's great book on predictions, "The Signal and the Noise: Why So Many Predictions Fail-but Some Don't":

A good model can be useful even when it fails. "It should be a given that whatever forecast we make on average will be wrong," Ozonoff told me. "So usually it's about understanding how it's wrong, and what to do when it's wrong, and minimizing the cost to us when it's wrong."

The key is in remembering that a model is a tool to help us understand the complexities of the universe, and never a substitute for the universe itself. This is important not just when we make predictions.

This storm tracking graphic is a model. Outside, the rain is falling and the wind is beginning to blow. That's reality.

Source: National Hurricane Center

Predicting the future is hard; ask a psychic

In the July-August 2007 Harvard Business Review, Paul Saffo explained in his article "Six Rules for Effective Forecasting" (link) that "prediction is possible only in a world in which events are preordained and no amount of action in the present can influence future outcomes." He goes on to write, "Above all, a forecaster's task is to map uncertainty, for in the world where our actions in the present influence the future, uncertainty is opportunity."

This belief was reinforced in an August 2007 news item from my local business newspaper, the Central Penn Business Journal:

...Business owners between 200-212 N. Second Street in Harrisburg will have to pack up if city officials approve plans for an 18-story office, parking and retail building at that location.

Four businesses would be affected: Cobalt Hair Salon, 2nd Street Psychic, Chilly Willy's ice cream parlor and the Tom Sawyer Diner....

Psychic Angie Tom, who owns 2nd Street Psychic, was in Europe and could not be reached. A psychic who works for Tom said she did not see this coming and could not say what Tom would do in the future.

If psychics can get caught off guard, then we must live in a world rife with uncertainty.

Dan Frommer shares one of his bad predictions

Predictions are fun to make and usually fun to forget. It's so easy to write about what you think will happen, and awkward or painful to look back and compare that to what really took place.

Dan Frommer of the cool SplatF blog took himself to task for an old prediction about how Palm and Flash could create a powerhouse mobile partnership ("The Dumbest Thing I've Ever Written About Flash"). Here's what he wrote in 2007 in Business Insider:

If Palm and Adobe could work together on a stunning user interface and offer the massive community of Flash developers wide-open access to a solid phone platform on good-looking devices, it could be a huge hit.

And here's the November 2011 view (after Palm had been purchased by HP and then set adrift, and Adobe announced that it would no longer develop Flash for mobile):

But it was so impractical! Not just the idea of Palm and Adobe banding together — they actually did try to work together on Flash for WebOS devices, and it still failed. But the idea of Flash working well on a mobile/touch device was so far-fetched in 2007, and is still pretty looney today. And that’s a big reason why Adobe is now winding down mobile Flash development.

Bad predictions are eight for a dollar in the tech world, but a sense of humor and self-reflection, as Frommer practices here, are much rarer commodities.

Scott Anthony cites "strategy gurus to avoid" - including himself!

In a great post on HBR.org, Innosight's Scott Anthony pricks the bubbles of highfalutin strategy gurus:

About a year ago I miscalculated — badly — on the Microsoft Kinect. In terms of speed of adoption, only Apple's iPad has rivaled the Kinect. Aside from some tough comments on my blog, the long-term repercussions were low. That's pretty much what happens to a pundit who gets it wrong — nothing.

And this:

The non-user. This pundit makes bold predictions about a product or service he's never used. Remember, a wide gulf separates a corporate press release or a carefully scripted demo and in-the-field reality. I ran into this trap when I started getting excited about Research in Motion's Playbookbefore the product had even been officially launched. The product's performance didn't live up to early press releases and has struggled in the market.

Finally, this:

The freeze framer. Back in 2005 I wrote a piece [PDF] looking at a new video recorder introduced by Pure Digital Technologies, the Flip. I said the simple, easy-to-use product had one fatal flaw — customers had to drop it off at drugstores to get videos processed. How many would do that, I wondered? The mistake I made was assuming that Pure wasn't already working on its second version, which featured a simple mechanism to load video onto computers. The company took off and was ultimately acquired by Cisco for hundreds of millions of dollars. (Cisco strangely shut down Pure Digital this year, but that's a different story.) Markets are dynamic, and if pundits don't take the long view they can make mistakes. 

How cool is it that Anthony is comfortable enough in his own skin to base a blog post on his bad predictions?