RESEARCH STANDARDS · UPDATED 22 OCTOBER 2026
Make the evidence inspectable.
A result is useful when its target, comparator, assumptions and limits can be understood and reproduced.
Evidence stages
| Stage | What should support the label |
|---|---|
| Exploratory | An observation or initial test; selection may be substantial. |
| Reproduced | Another execution of the method with recorded inputs; independence must be documented. |
| Out-of-sample | Frozen decisions evaluated on a chronologically reserved sample. |
| Demo forward | Prospective execution observations on a demo account. |
| Live | Real-account observations with costs and cash-flow treatment recorded. |
Labels are author-reported evidence descriptions, not certifications. Live observations do not guarantee future performance. A changed strategy needs a new validation record.
Forecast diagnostics
Probability: Brier score is the mean squared difference between predicted probability and a binary outcome. Log loss penalizes confident mistakes. Reliability tables compare mean forecast probability with observed frequency. A lower Brier score alone does not isolate calibration improvement.
Magnitude: MAE, RMSE and prediction-minus-observation bias use the stated target units. Interval coverage counts observations within inclusive bounds and must be interpreted alongside nominal coverage and interval width.
Ranking: Spearman correlation is Pearson correlation of average-tie ranks. Pooled correlation combines all supplied observations; it is different from averaging independently defined cross-sectional IC groups. Constant ranks have undefined correlation. No automatic significance claim is made.
Timing: Probability of an event within a specified horizon needs a mechanical timing baseline and explicit label availability. Horizons and overlapping samples must remain visible.
Regimes: Confusion matrices, class precision, recall and macro F1 expose class imbalance. Contemporaneous classification does not by itself establish the ability to predict transitions.
Wilson intervals describe a binomial proportion under independent Bernoulli sampling. Overlapping labels and clustered returns require an appropriate dependence-aware analysis. Comparisons use rows with both model and baseline forecasts.
Validation
Freeze the hypothesis, event definition, baseline, primary metric and rejection criteria before final testing. Reserve chronological holdouts, account for label overlap at split boundaries and document every tested variant. Ordinary random splits can leak future information in time-series work.
Inspect rolling and regime-conditioned results, sample sizes, parameter sensitivity and multiple-testing risk. Walk-forward selection must occur inside each training window. A final holdout loses its status when repeatedly used to change the model.
From forecasts to execution
Forecast skill does not establish profitability. A strategy needs executable entry, exit, sizing, costs, financing where relevant and a realistic fill model. Same-bar ambiguity must be resolved with finer data or explicitly reported. Report simulated, demo and live results separately.
Return frequencies and cash-flow conventions must match annualization assumptions. R-based trade statistics are not automatically account-level Sharpe ratios. Portfolio blending in weighted R is a research model, not a funded equity simulation.
Reproducibility and revisions
Preserve dataset identifiers, code revisions, parameters and exact inputs. YDLib record revisions preserve submitted content. Test-lab input archives preserve the supplied bars and configuration; they do not independently certify browser-computed outputs. Document the reason for corrections and inspect connected strategies when evidence changes.
Run the public methods study →Primary references
- Probability calibration · scikit-learn
- Spearman correlation · SciPy
- Mean pinball loss · scikit-learn
- Time-series validation · scikit-learn
- WCAG 2.2 · W3C
These references support method definitions. They do not endorse YDLib or validate a trading strategy.