Superforecasting: The Future Of Analyst Performance Measurement

Anyone that buys a stock is a forecaster. The only reason to buy a stock is because you believe the value of the stock will be greater in the future than it is today. That is the definition of a forecast. But how do you know if you’re any good? How do you measure your stock forecasts? And how can you use those measurements to improve your forecasts in the future? Answering these questions is the next mission of Alpha Theory. A vision that was crystalized by the illuminating book “Superforecasting”.

P&L is the primary measure of forecasting acumen used today. But making money on an investment is only a crude measure of forecasting skill. It can be corrupted by trading decisions, sizing decisions, liquidity constraints, etc. Positive P&L can also mask bad process and outlier P&L observations can overwhelm the data and convince you that luck is actually skill. Forecasting skill should be measured by actually measuring price forecasts. That seems obvious, but there are very few managers actually doing it.

If it isn’t obvious, the primary reason for a better forecasting measure is so that you can improve. There are hundreds of psychology, intelligence, and physical training studies that show how important good feedback is to making improvements (Feedback | Mundanity of Excellence). Right now, the investing community operates like a golfer trying to improve his swing by practicing at night with no lights. He can feel if he made good contact and has a general sense of direction, but doesn’t know how far his ball landed from the target. Turn the lights on and the golfer gets immediate feedback that allows him to improve rapidly. I believe that will happen in investing for those willing to “turn the lights on” by making price target forecasts a required part of their process.

What is required? The first step is getting explicit price forecasts from your analysts. The ideal way is to ask for a range of potential outcomes with probabilities. This allows your analysts to describe the range of possibilities that come from their research. Step two is to measure the forecast in a way that allows you to compare analysts. Step three involves creating feedback that helps refine and improve your process.

Step One: Capture Price Forecasts

A system that stores a history of all price targets and allows analysts to make frequent updates is required (this statement is clearly self-serving given what Alpha Theory does, but it’s a requirement none the less). The required inputs are price targets and probabilities that add to 100%. Additional nice to haves are a time horizon for price targets and rationale for how the price targets were derived.

Step Two: Measure the Forecast

The plan is to use a modified Brier Score.

f = probability of outcome

o = outcome (1 if occurs, 0 if it doesn’t)

Brier Scores were designed to measure probability-based binary outcomes. For example, assume I forecast an 80% chance that Donald Trump is the GOP Presidential Candidate in 2016. The math works like this.

BS if not nominated = (0.8 – 0)^2 = 0.64

BS if nominated = (0.8 – 1)^2 = 0.16

Brier Scores are like golf scores, lower is better. The worst score is a 1 and the best is a 0 (100% probability forecast and you get it right = 0). The issue with simple Brier Score is that it assumes binary outcomes (i.e. Trump is the candidate or he is not). It also doesn’t account for “close” to right (it doesn’t matter if you were close to right forecasting Trump was the candidate but it is important if you were close to your price target forecast). The Alpha Theory team is actively working to develop an adapted Brier Score to measure forecasting acumen. We welcome any and all feedback as we tackle this challenge.

Step Three: Use Feedback to Improve

Over the past five years, Alpha Theory has accumulated over 10,000 price target forecasts from over 500 buyside analysts. With this data, we will test our measurement techniques and give our clients’ forecast assessments and feedback loops required for improvement. Feedback comes in many different forms. Immediate feedback will be notifications when prices meet or exceed price forecasts. The notifications will include the date of the original forecast and the rationale for the target. Intermediate feedback will be the measured difference between the actual and forecast price at the peak, trough, and terminal price. Finally, after an analyst has made a statistically significant number of forecasts, a modified Brier Score will incorporate both price target and probability forecasting acumen.

The future of forecasting involves better measurement. Political pundits, economic analysts, meteorologist, and every other profession that gets paid to make forecasts should have their forecasting score right beside their forecast. Imagine you’re watching CNBC and a floor trader comes on and says, “I believe there is a 70% chance the S&P breaks 2,000 in the next couple of weeks.” And right beside their name you see a Forecasting Score of 0.61 and realize that the forecast is slightly worse than a coin flip. It is much easier for pundits to operate in a world where statements are rarely judged. I think we’d all love that world. If you’re a portfolio manager, you have the power to create a forecasting world where track records and performance are measured and matter. Capture price targets with probabilities, keep a history, and measure accuracy.