Machine learning in trading

Definition

Machine learning in trading applies statistical learning — from regularized regressions through gradient-boosted trees to deep networks — to prediction, signal combination, and execution problems. The KB carries it with a specific honest framing (Lopez de Prado’s, the field’s standard reference): financial data is uniquely HOSTILE to ML — tiny effective sample sizes, non-stationary regimes, near-zero signal-to-noise, and adversarial adaptation (philosophy-adaptive-markets — profitable patterns get arbitraged BECAUSE they’re found). ML amplifies both genuine signal extraction and every hygiene failure in quant-backtest-hygiene — it is a power tool that mostly finds leakage faster.

How it works / structure

• Where ML earns its keep (documented use classes): signal COMBINATION (nonlinear aggregation of many weak features — the strongest evidence base, e.g. Gu-Kelly-Xiu’s asset-pricing-with-ML results); unstructured-data extraction (text/NLP for sent-news-social pipelines); execution optimization (fill/impact prediction on abundant microstructure data — the sample-size problem is mildest here); regime classification as probabilistic tagging.
• The financial-data pathologies (Lopez de Prado): observations are SERIALLY DEPENDENT (standard cross-validation leaks — purged/embargoed CV is the correction); labels overlap in time; regimes make train/test exchangeability false; and effective history is short (decades of daily data is a few thousand points — deep-learning appetites vs a starvation diet).
• The discipline stack (engine-relevant): feature/label engineering over architecture (the documented practitioner consensus — problem formulation beats model choice); regularization and ensembling as defaults; feature-importance stability across time as the sanity check; TRIAL ACCOUNTING inherited from quant-backtest-hygiene (hyperparameter searches are thousands of implicit backtests); interpretability requirements scaled to position size (unexplainable models get sandbox capital, labeled practitioner rule).
• The adversarial ceiling: unlike vision or language, markets ADAPT to their predictors — edges decay on discovery (McLean-Pontiff decay applies with force); ML systems need decay monitoring and retirement protocols, not just deployment gates.

When it applies

Signal-combination layers over KB-derived features (the engine’s natural use — structured entries as feature vocabulary); text/sentiment ingestion; execution-quality modeling; NOT as an oracle for “what will the market do” — the KB’s framing is ML as component within falsifiable theses, never as a substitute for them.

Risk profile & failure modes

• Leakage supremacy (the field’s documented normal): most spectacular ML-trading results trace to look-ahead, label leakage, or survivorship (quant-data-hygiene) — the model’s power makes contaminated data look like alpha with high confidence.
• Non-stationarity betrayal: models fit to one regime fail silently at transitions — exactly when positioning matters most; regime-conditional validation and live decay monitoring are mandatory.
• Complexity worship: deeper models on the same starved data mostly fit noise better — the documented simple-beats-complex base rate in low-signal domains; complexity requires sample-size justification.
• Interpretability debt: unexplainable positions can’t be risk-checked against theses — a governance failure independent of accuracy (the engine’s falsifiability rule applies to model-driven claims too).

Evidence & limits

Lopez de Prado (2018) codifies the methodological corrections; Gu-Kelly-Xiu (2020, RFS) is the peer-reviewed benchmark for ML asset-pricing gains; decay evidence is documented. The field’s honest summary: real but modest gains for disciplined teams, catastrophic self-deception for undisciplined ones — the differentiator is hygiene, not architecture.

Falsifiable-thesis examples

Illustrations only, not signals:

• “Model X’s live information coefficient stays within 50% of its purged-CV estimate over 6 months (validity-transfer check)” — falsified by the live IC series.
• “Feature-importance rankings remain rank-correlated

  0.6 across yearly refits (stability check)“ — falsified by the importance drift.

Cross-references

• The hygiene foundations: quant-backtest-hygiene, quant-data-hygiene
• The theory frame: philosophy-adaptive-markets
• The application seams: strategy-factor-investing, sent-news-social