Why backtests lie

October 14, 2026·3 min read

Somewhere right now, someone is staring at a backtest showing 34% annualized returns with a Sharpe ratio of 2.1, and they believe it. They should not. A backtest is not evidence that a strategy works. It is evidence that a strategy worked on data the researcher already saw — which, without extreme discipline, is evidence of nothing at all.

Overfitting, concretely

Every parameter you tune — the lookback window, the entry threshold, the stop-loss distance — is a degree of freedom spent fitting the past. With enough parameters, you can fit any historical dataset perfectly, including pure noise. This is not a metaphor: given a random walk, a researcher trying a few dozen parameter combinations will routinely “discover” a strategy with an impressive Sharpe ratio. The strategy learned the noise, not a signal.

The tell is complexity. A strategy with eleven tuned parameters that beats the market by 4% a year is almost certainly an overfit. A strategy with one parameter and an economic reason it should work deserves at least a hearing. Simplicity is not aesthetic preference here; it is degrees-of-freedom accounting.

The multiple-comparisons problem

Here is the deeper issue. Suppose you test 100 random strategies on 20 years of data. At a 5% significance level, you expect about five of them to look statistically significant by pure chance. The researcher then publishes the best one — the one that survived a selection process designed to find flukes. This is why published backtests cluster suspiciously just above “significant.”

The finance literature has a name for the graveyard of strategies that worked until publication: most of them. The honest correction is brutal — divide your significance threshold by the number of things you tried (the Bonferroni correction), or better, demand out-of-sample confirmation on data the strategy never saw during development. A backtest without a holdout period is a rough draft, not a result.

How to read a backtest without fooling yourself

Four questions. One: how many variations were tried before this one? If the answer is “a few,” multiply your skepticism accordingly. Two: is there an economic reason this should work, or only a statistical pattern? Patterns without reasons are usually noise wearing a costume. Three: what happens on out-of-sample data and in different market regimes? A strategy that only works in bull markets is a bull market with extra steps. Four: are transaction costs, slippage, and taxes included? Many beautiful backtests die the moment you pay to trade them.

The base rate is grim: the overwhelming majority of backtested edges decay or vanish. Start from the assumption that a backtest is lying, and make it prove otherwise. That single habit will save you more money than any indicator.