7 steps to mastering the business of super forecasting
ECONOMICS STRATEGY 01 Mar 2016
Philip Tetlock put forecasters under the microscope to distinguish between the good and the great
Predicting the future isn’t easy, especially in business. But Canadian psychologist Philip Tetlock says it’s a skill that can be cultivated to provide a more accurate picture of what lies ahead.
Anyone who stays in business will face the problem of trying to see into the future. When you think about hiring a new employee, you ask: “Will we have enough new business to keep them occupied?” When you pitch for a new piece of business, you think: “How many hours will it take us to complete this job?” You have to forecast.
Forecasting can seem like a dark art. Psychologist and political scientist Philip Tetlock insists not only that it is a skill, but also that you can get better at it. Over recent years, he has spent his time creating teams of forecasters that compete in – and win – forecasting tournaments funded by the US Government intelligence community. In the process, he has become perhaps the world’s leading expert on predicting the future.
Now Tetlock has set out his forecasting techniques in a book, Super forecasting: The Art and Science of Prediction, co-authored with journalist Dan Gardner.
Here are seven steps that Tetlock suggests can drive better forecasts – steps that any organisation can take to deal with its own forecasting challenges.
1. Break seemingly intractable problems into tractable sub-problems.
Super forecasters split problems into knowable and unknowable parts. They flush ignorance into the open.
“Expose and examine your assumptions,” says Tetlock. “Dare to be wrong by making your best guess. Better to discover errors quickly than to hide them behind vague verbiage.”
One of the surprising lessons we learn from the best forecasters is that very often “remarkably good probability estimates arise from a remarkably crude series of assumptions and guesstimates”.
“Forecasting is a serious skill... if you take it seriously, you can cultivate it and you can get better at it.” Philip Tetlock
The physicist Enrico Fermi (designer of the first nuclear reactor) understood this. He used to love posing seemingly impossible questions like “How many piano tuners are in Chicago?”. Then he’d chop up the problem. How many people are there in Chicago? How many will own pianos? How often do they get them tuned? How many pianos can one person tune in a day? Rough estimates of the answers to these questions can get you an answer surprisingly close to the right figure.
2. Strike the right balance between inside and outside views.
Super forecasters know that there is nothing new under the sun.
“Uniqueness is a matter of degree,” says Tetlock. Super forecasters conduct creative searches for comparisons even for seemingly unique events, such as the standoff between a new socialist government in Athens and Greece’s creditors.
Forecasting can seem like a dark art but being able to see into the future in business situations is a skill that can be honed over time.
“Super forecasters are in the habit of posing the outside-view question – ‘how often do things of this sort happen in situations of this sort?’.”
3. Strive to distinguish as many degrees of doubt as the problem permits – but no more.
While few things are either certain or impossible, “maybe” is not an informative response. Tetlock says nuance matters.
“The more degrees of uncertainty you can distinguish, the better a forecaster you are likely to be,” he says.
“Translating vague-verbiage hunches into numeric probabilities feels unnatural at first but it can be done. It just requires patience and practice. The super forecasters have shown what is possible.”
Another way of putting this is that at the poker table, the player who can distinguish 45/55 bets from 50/50 bets will probably wins.
4. Strike the right balance between under-reacting and overreacting to evidence.
Tetlock says updating your beliefs is as important to good forecasting as brushing and flossing is to good dental hygiene. It can be boring, occasionally uncomfortable, but it pays off in the long term.
“Skilful updating requires teasing subtle signals from noisy news flows, all the while resisting the lure of wishful thinking,” he says, adding that super forecasters tend to update their beliefs in small increments, often moving from probabilities of, say, 0.4 to 0.35. These distinctions are too subtle to capture with vague verbiage like “might” or “maybe”, but in the long run, Tetlock says these distinctions define the difference between good and great forecasters.
5. Look for the errors behind your mistakes, but beware of rear-view mirror biases.
Rather than trying to justify or excuse your failures, Tetlock says you should own them and conduct unflinching post-mortems to examine where you went wrong.
“Remember that although the more common error is to learn too little from failure and to overlook flaws in your basic assumptions, it is also possible to learn too much. You may have been basically on the right track but made a minor technical mistake that had big ramifications.”
He suggests doing post-mortems of your successes, too. “Not all successes imply that your reasoning was right. You may have lucked out by making off-setting errors. If you keep reasoning along the same lines, you are setting yourself up for a nasty surprise.”
6. Bring out the best in others and let others bring out the best in you.
Master the fine art of team management. This includes understanding the arguments of the other side, precision questioning so you can help others clarify their arguments, and learning to disagree without being disagreeable.
“Wise leaders know how fine the line can be between helpful suggestions and micro-managerial meddling … or between a scatterbrained group and an open-minded one,” says Tetlock.
7. Focus on questions where your hard work is likely to pay off.
Certain classes of outcomes, such as oil prices and currency markets, are notoriously hard to predict. Tetlock says we usually don’t know how unpredictable outcomes are until we’ve spun our wheels trying to gain analytical traction.
“There are two basic errors it is possible to make here,” he says. “We could fail to try to predict the potentially predictable or we could waste our time trying to predict the unpredictable. Which error would be worse in your situation?”
This article was originally published on intheblack.com and is reproduced with kind permission from CPA Australia.
