Backtesting Pitfalls: Overfitting, Survivorship Bias and Look-Ahead

A backtest replays a trading rule on historical data to see how it would have performed. It is a useful filter, but it is also remarkably easy to fool. Many impressive equity curves come from the test setup rather than from the idea. Knowing the common traps helps you tell a promising result from a flattering one.

What a backtest can and cannot tell you

A backtest answers a narrow question: given these rules, this data and these assumptions about costs and fills, what would have happened? It cannot say what will happen. At best, it can reject weak ideas cheaply and give a rough picture of how often a method loses, how deep its drawdowns went and how sensitive it is to settings.

Because you choose the data, the rules and the assumptions, every choice is a chance for bias to creep in. The five pitfalls below account for most of the gap between backtested and live results.

Overfitting: tuning to noise

Overfitting, also called curve fitting, means shaping a strategy so closely to past data that it captures random quirks instead of repeatable behaviour. The more adjustable settings a rule has, the easier it is to do.

Here is a scale example. Say you test moving-average lengths from 5 to 100 in steps of 5 (20 values), stop sizes in 20 steps and 10 different profit targets. That is 20 × 20 × 10 = 4,000 combinations. Even if every one of them were a coin flip with no real edge, a handful would look excellent by luck alone. Picking the winner of 4,000 random tries tells you which one was luckiest, not which one is best.

Survivorship bias: the missing losers

Survivorship bias appears when the dataset only contains assets that survived to today. Companies that went bankrupt, were delisted or were acquired at a loss are quietly absent, and so are their losses. Testing a share-picking rule on today's list of index members is a classic case.

Illustrative numbers show the effect. Suppose 100 shares were available at the start of a ten-year period. Eighty survived and earned an average of 10% a year. Twenty were delisted and lost an average of 15% a year. The full group averaged (80 × 10 + 20 × −15) ÷ 100 = 5% a year. A test using only the survivors reports 10%, double the truth. The same problem can affect funds, trading systems with published records and even currency pairs that were dropped from a platform.

Look-ahead bias: knowing the future by accident

Look-ahead bias means the test uses data that would not have been available at the decision point. It is often subtle. Examples include buying at the closing price of a bar after the signal was calculated from that same close, using the day's high or low to place an order earlier that day, or working with economic figures in their later revised form instead of the version published at the time.

• Trade on the next bar's open, not on the bar that created the signal, unless you can genuinely act at the close.
• Check that every indicator uses only past values, particularly custom or repainting indicators, which change their historical plots after the fact.
• Use point-in-time data for fundamentals and for index membership.
• Be sceptical of any result that looks too smooth. Real trading rarely produces perfectly steady curves.

Costs, fills and sample size

A rule that earns a small edge per trade can disappear once costs are added. Intraday methods with many trades are the most exposed, because each trade pays the spread again. Model costs conservatively, use realistic spreads for the time of day, and add slippage for stop orders, which usually fill at a worse price than their level during fast moves.

Sample size matters as much. A backtest with 25 trades is an anecdote. Many practitioners look for at least a few hundred trades, spread across different market conditions, before treating the statistics with any confidence. Conditions also matter: a test run only through a long steady uptrend says little about a ranging or falling market.