Backtest hygiene & overfitting

Definition

Backtest hygiene is the discipline that separates evidence from artifact when testing strategies on historical data. Its necessity is proven, not asserted: Bailey et al (2014) demonstrated mathematically that trying enough strategy variants GUARANTEES finding one with a spectacular backtest even on random data — and that the standard practice of reporting only the best variant makes the result meaningless without knowing how many were tried. Harvey-Liu-Zhu (2016) applied the same logic to academia itself: after accounting for the hundreds of factors tested, most published anomalies fail appropriate significance thresholds (they propose t > 3.0, not 2.0). Every backtest number the engine consumes must carry its hygiene provenance.

How it works / structure

• The core failure (multiple testing): N trials of random strategies produce a best-of-N Sharpe that grows with N — the documented expected maximum; a “2.0 Sharpe” means nothing without N; Bailey et al’s Deflated Sharpe Ratio and Probability of Backtest Overfitting (PBO) are the correction machinery, and the minimum honest disclosure is the TRIAL COUNT.
• The canonical sins (each checkable): LOOK-AHEAD (using information unavailable at decision time — point-in-time data is the cure, quant-data-hygiene); SURVIVORSHIP (testing on today’s index members); in-sample optimization without out-of-sample holdout (and holdout REUSE, which silently converts out-of-sample back to in-sample); cost/slippage omission (documented to erase most high-turnover anomalies); regime cherry-picking (start/end date selection as a hidden parameter).
• The hygiene protocol (engine-relevant): pre-registered hypothesis before the test (disc-process-vs-outcome); parameter counts and trial logs kept; walk-forward or purged cross-validation rather than single splits; costs modeled at realistic aggression; robustness maps (performance across parameter neighborhoods — cliffs mean overfit) rather than point estimates; Deflated Sharpe reported when trials > 1, which is always.
• The interpretive standard: a strategy that survives hygiene is a CANDIDATE with an estimated edge and wide error bars — the documented out-of-sample haircut (McLean-Pontiff: published anomalies decay ~50%+ post-publication) is the base rate for how much backtests flatter.

When it applies

Every quantitative claim in the KB and every strategy evaluation the engine performs — this entry is the gate, not an option; factor claims (strategy-factor-investing zoo skepticism is this entry applied); indicator threshold choices (every “works above X” inherits multiple-testing risk); simulation design and vendor/backtest-report consumption (the trial count question first, always).

Risk profile & failure modes

• The N you don’t see: the deadliest form is UNCOUNTED trials — every discarded variant, every “quick look” that shaped the next test; honest N is larger than logged N, so deflation is a floor, not a fix.
• Holdout exhaustion: each peek at the test set spends it — teams that iterate against one holdout converge on overfit with clean-looking process (documented in the ML-finance literature).
• Hygiene theater: walk-forward with re-optimized everything, robustness checks run until they pass — process vocabulary without its substance; the KB’s test is whether a FAILED result would have been accepted and recorded.
• Over-correction paralysis: t > 3 thresholds and full deflation reject nearly everything — the documented tension; the honest posture is graded confidence with disclosed hygiene, not binary accept/reject.

Evidence & limits

Bailey et al (2014) and Harvey-Liu-Zhu (2016) are the peer-reviewed anchors; McLean-Pontiff’s decay evidence quantifies the flattery. The machinery corrects KNOWN trial counts — unlogged exploration remains the irreducible honesty problem, which process discipline addresses and mathematics cannot.

Falsifiable-thesis examples

Illustrations only, not signals:

• “Strategy X’s live 12-month Sharpe lands within its deflated-Sharpe confidence band (hygiene-validity check)” — falsified by live performance outside the band.
• “Parameters within ±20% of chosen values retain >70% of backtest performance (robustness-cliff test)” — falsified by the parameter-neighborhood map.

Cross-references

• The data substrate: quant-data-hygiene; the ML extension: quant-ml-in-trading
• The process frame: disc-process-vs-outcome
• The applied domains: strategy-factor-investing, risk-scenario-analysis