METHOD STUDY · SYNTHETIC DATA · VERSION 1
Better forecasts.
Different decisions.
Why probability quality, classification accuracy and trading expectancy must be evaluated separately.
YDLib Research · Published 22 October 2026 · 6 minute read
This is an executed synthetic experiment, not a market backtest. It establishes no trading edge.
Question
Can two forecasts have the same classification accuracy and different probability scores? And can a useful forecast still produce poor results under a chosen decision rule and cost?
Design
We generated 2,000 independent binary outcomes from probabilities sampled uniformly between 0.2 and 0.8, with fixed seed 20261022. We compared the known generating probability, an overconfident forecast shifted 0.18 away from 0.5, and a fixed 0.5 baseline. All forecasts were scored on exactly the same outcomes.
Knowing the generating probability is intentionally unrealistic: it isolates evaluation concepts without claiming a forecasting model has discovered market information. The dates are synthetic row labels, not trading sessions.
| Forecast | Brier | Log loss | Accuracy at 0.5 |
|---|---|---|---|
| Generating probability | 0.2191 | 0.6286 | 65.0% |
| Overconfident | 0.2509 | 0.7386 | 65.0% |
| Fixed 0.5 | 0.2500 | 0.6931 | 48.1% |
Calibration and sample counts
| Probability bin | Mean prediction | Observed frequency | Sample |
|---|---|---|---|
| 20–30% | 24.9% | 26.9% | 327 |
| 30–40% | 35.0% | 29.6% | 345 |
| 40–50% | 45.1% | 43.5% | 363 |
| 50–60% | 55.2% | 50.6% | 324 |
| 60–70% | 64.9% | 65.1% | 312 |
| 70–80% | 75.2% | 75.1% | 329 |
A hypothetical decision and its costs
For illustration, take only rows with generating probability above 0.55. Assign +1R for an event and −1R otherwise, then subtract a fixed cost per selected row. This artificial payoff is not a fill model or a market strategy.
| Cost per decision | Selected outcomes | Mean net R |
|---|---|---|
| 0.00R | 823 | 0.3147R |
| 0.05R | 823 | 0.2647R |
| 0.15R | 823 | 0.1647R |
| 0.30R | 823 | 0.0147R |
| 0.40R | 823 | -0.0853R |
Interpretation
The overconfident and generating-probability forecasts make the same binary classifications at 0.5, yet their probability scores differ. Changing the assumed cost changes net expectancy without changing either forecast. This is why a signal report and a strategy report need different diagnostics.
Limitations
The outcomes are independent, the generating probabilities are known, and there is no execution uncertainty, selection search, capital constraint, drift or market impact. Real financial labels often overlap. The result demonstrates metric definitions; it cannot estimate financial performance.
Reproduce the experiment
The dependency-free JavaScript file contains the generator, scoring rules, bins and cost scenarios. Download it, inspect it, and run node forecast-score-study.mjs in Node.js. The seed and computation are fixed.
References: scikit-learn probability calibration and YDLib methodology.