American Idols

I was lucky enough to be part of a small event, The Frontier of Forecasting Conference, hosted by Good Judgment Inc. Among the participants were Phil Tetlock, Barbara Mellers, and Daniel Kahneman. For those that don’t know, Kahneman is a Nobel Laureate and considered the father of Behavioral Economics. Tetlock and Mellers are the brains behind Superforecasting.

Several of you were interested in attending but unable to make the trip. The following is a summary of the presentations from the conference.

‍

Tetlock on Gisting

Good Judgment's CEO Terry Murray opened up the morning by introducing the founder of the company, Phil Tetlock. Phil talked about a new idea that he’s working on called Gisting. The goal is to improve understanding by taking a large amount of information and having multiple people create a Gist or a shorter explanation of the information. These Gists would then be graded by peers and the best ones would be picked and synthesized into a team Gist. This leads to a deeper understanding by the “gisters” and easier understanding by readers that only have time for the gist.

Gists are important because full understanding is never just quantitative or qualitative. Superforecasting is quantitative. Supergisting attempts to provide the qualitative piece. The challenge is that time is scarce and this is a new task that will meet resistance in most organizational culture.

Gisting is a relatively new idea and it will be interesting to watch how it develops as Phil, Barbara, and Good Judgment group put more time into research. The next book, Supergisting?

‍
Kahneman on Noise

Kahneman was next up and he spent his time talking about Noise. The concept is not new but he believes it should become a focus because it is easier to reduce than bias. He described an insurance company that he worked with to improve claims adjuster accuracy. He measured the efficacy of their claims process by having independent adjusters price the same claim. The average difference in claim value was 50%! That means that one adjuster might write a check for $1,000 and another for $1,500 for the same claim. He described how a simple algorithm would dramatically reduce noise and improve claim accuracy.

The discussion took a slightly cynical tone when he described how few of his practical ideas were actually put into practice. For example, the insurance firm, after learning of these gross miscalculations, didn’t implement the systematic approach he suggested. He gave another example of how Steven Levitt, of “Freakonomics” fame, showed a simple system of fraud detection improvement to a credit card company that would have saved many millions a year, but wasn’t implemented.

Kahneman said, “change causes winners and losers. Losers are much louder than winners, which makes reform much less likely.” And that “leaders don’t want to see their mystique questioned by systems.” Dr. Mellers had a nice rejoinder that “things will change, one funeral at a time.” For all of us Superforecasting believers, we hope it happens faster than that.

I believe the success that Ray Dalio and Bridgewater have seen by being very systematic and process-oriented may shed some light and make leaders less resistant to change. The publishing of Dalio’s “Principles” will be read by many leaders and get a conversation started about how we all can improve by being more disciplined.

Idea Exchange on Forecasting

The second half of the day was a “safe zone” event to permit free-flowing exchange of ideas due. This means that I’m not allowed to comment on the dialog but I can give a high-level recap.

I was a panelist for “Improving Probabilistic Forecasting Within Organizations.” The goal was to give real world examples of people implementing forecasting tools to improve decision making. It was exciting to see many firms experimenting with forecasting systems. In my view, shared by Good Judgment's president Warren Hatch, who chaired the panel, the challenge that most faced was getting broad adoption and keeping momentum.

The critical component for solving this challenge is getting top-level buy-in. If senior leadership asks questions and uses the output to make decisions, then people will participate. Another strategy for increasing participation was active feedback. Providing scores, leaderboards, best/worst forecasts, stats, etc. have a demonstrable impact on usage.

‍
Better Forecasting Through Better Models

The final discussion was “Bayesian Cluster Forecasting Models for Strategic Decision-Making” lead by Dr. Kathryn McNabb Cochran. She is part of Good Judgment Inc. and is a leader in the field of better decision making through forecasting. The goal is to make better forecasts by creating better models. The models are a hybrid of pure forecasts and adjustments that lead to more accurate forecasts.

For anyone curious about how they can be better forecasters and apply that thinking to their organization, please contact the great folks at Good Judgment Inc.

Final Thought

Meeting several of my heroes in one day made me think how nice it would be if the GE ad campaign in which great scientists are treated like stars was reality. How cool would it be if my girls could grow up in a world where Kahneman and Tversky were admired as much as Brady and Gronk.