AI forecasting has a scoreboard now, and the models are climbing it fast. That is genuinely useful. It also hides the part of forecasting that matters most for anyone tracking companies: knowing what to forecast, and knowing it early.
What ForecastBench measures
ForecastBench, from the Forecasting Research Institute, is the most serious attempt so far to measure how well AI systems predict the future. Its design solves the obvious problem with testing a language model on forecasting, which is that the model may already have read the answer:
- Only unresolved questions. Every question is about something that has not happened yet when the forecast is made, so nothing can be memorised.
- A rolling question set. Around a thousand questions, refreshed regularly, drawn from forecasting platforms and prediction markets and generated automatically from public datasets.
- Human baselines. Models are compared with the general public and with superforecasters, people with a long, measured record of accurate forecasts.
- A proper score. Forecasts are scored with the Brier score: the squared distance between the probability given and what actually happened. Lower is better, and confident wrong answers are punished hard.
At the time of writing, superforecasters still lead, with a Brier score of about 0.086 against about 0.103 for the best models (FRI). The gap has been shrinking with every model generation.
Where the benchmark stops
None of this is a flaw in ForecastBench. A benchmark has to fix the questions in order to compare forecasters fairly. But fixing the questions removes the four things that make forecasting company moves hard.
1. The question is handed to you
Every ForecastBench question arrives already written: a clear event, a date, a way to resolve it. In business nobody hands you "Will this competitor open a sales office in Germany by June?" Noticing that this is a question worth asking is most of the work. A system that answers well-posed questions brilliantly but never poses one has solved the easier half.
2. The questions are already in the spotlight
Questions that come from prediction markets and public datasets are, by construction, about things people are already watching: elections, interest rates, big-tech launches. The moves that create advantage are the ones nobody has put a market on yet. A mid-sized company's registry filing in Oslo, or a cluster of job adverts in a city where it has no office, will never appear on a market until it is old news.
3. Being early isn't scored
A Brier score rewards being right, not being first. In competitive intelligence the value of a forecast falls fast as the announcement gets closer. Calling a move nine months ahead is worth far more than calling it the day before the press release, even if both are equally accurate. A benchmark with a fixed question set has no way to credit lead time over everyone else.
4. One number hides the reasoning
Real evidence is ambiguous. A burst of hiring might mean expansion, replacement of people who left, or a project that is about to be cancelled. Intelligence analysts have a name for the discipline of handling this: analysis of competing hypotheses, set out by Richards Heuer at the CIA. You keep several explanations alive, and you look for the evidence that tells them apart, rather than the evidence that fits the story you already like. A benchmark scores the final probability, not whether it moved for the right reasons.
What a benchmark for early signals would score
If you wanted to measure forecasting the way a strategy or sales team needs it, the questions could not be fixed in advance. Instead you would score:
- Discovery. Did the forecaster raise the question before it became public? A prediction of something already announced is worth nothing.
- Lead time. How long before the announcement did the forecast appear, and how high was the probability at that point?
- Calibration on its own questions. The Brier score still applies, but to the questions the forecaster chose to ask.
- Difficulty. A forecaster who only asks easy questions will look well calibrated. Questions should be scored against what was publicly known when they were asked.
- False alarms. How many of the moves it flagged never happened, and did it say so as the evidence turned?
This is harder to run than ForecastBench, because the forecaster writes its own exam. But it is much closer to the job.
Why this matters
As models get better at answering well-posed questions, answering stops being the bottleneck. The scarce skill becomes noticing early that something is moving, turning it into a question that can be checked, and keeping track of the evidence for and against as it comes in. That is the part of forecasting Paldo is built for.
You can watch it happen on our live predictions about Tesla, Microsoft, Amazon, AMD and NVIDIA, each with a likelihood, a deadline and the evidence on both sides, updated as research runs. For the moves behind predictions like these, see the public company moves pages.