Backtesting Lies: Why Backtests Overstate Results
Educational content, not financial advice. This post contains no performance figures from our own backtests or paper trading.
Backtests flatter strategies. Here's why backtests lie, what they don't capture, and how to use them without getting fooled.
Uncomfortable truth: If your backtest looks too good to be true, it's probably lying to you. Here's why, and what to do about it.
The Backtest That Looks Perfect
Almost every strategy looks good in a backtest, because the test is run on history that you have already seen. The smoother the equity curve, the more suspicious you should be. Backtest results routinely overstate what live or paper execution delivers.
We are not publishing numbers from our own backtests or paper trading in this post. Here is what backtests commonly leave out.
Lie #1: No Slippage Modeling
A backtest that assumes you get the exact close price ignores slippage, the gap between the price you expected and the price you got. It matters most on market orders, in thin order books, and on low-liquidity prediction markets, and it compounds across many trades.
Lesson: model slippage explicitly, and use larger assumptions for thin markets.
Lie #2: Perfect Execution
A backtest assumes a signal fires and executes at the same instant. In practice, API rate limits, latency, and outages delay execution, and the price can move in the meantime.
Lesson: add latency and rate-limit assumptions to your simulation rather than assuming instant fills and infinite liquidity.
Lie #3: Regime Assumptions
A backtest period dominated by one kind of market, such as a long range-bound stretch, flatters strategies that suit that regime. Mean reversion tends to do well in ranges and poorly in strong trends. A single headline number hides that dependence.
Lesson: segment backtests by market regime. If a strategy only suits one regime, label it as such and do not run it in the other.
Lie #4: No Psychological Pressure
A backtest keeps trading through any losing streak with robotic discipline. Humans running a live system want to intervene after a run of losses, and that urge can lead to bad decisions.
Lesson: if you build an automated system, code in the pauses and circuit breakers you will want, then backtest with those included.
Lie #5: Overfitting to History
It is easy to tweak thresholds and add filters until a backtest looks great. Parameters tuned this way fit the historical period, with no guarantee about the future.
Lesson: if you are adjusting parameters to improve backtest results, you are probably overfitting. Use walk-forward analysis: optimize on one period, then test on unseen data.
What Backtests Are Actually Good For
Backtests are not useless. They are just not predictive. They are useful for:
1. Sanity Checking
If a backtest looks implausibly good, the logic is probably wrong. Backtests catch obvious bugs.
2. Strategy Comparison
Even if absolute numbers are inflated, comparing two strategies under the same assumptions can still be informative.
3. Ruling Out Bad Ideas
If a strategy has no positive expectancy even in a generous backtest, it almost certainly has no edge. Backtests are better at ruling out bad ideas than confirming good ones.
4. Parameter Sensitivity
Test a range of parameter values. If performance swings wildly, the strategy is fragile. If it is stable across a range, that is a better sign. Robustness matters more than peak performance.
A Sensible Validation Sequence
A reasonable sequence moves from cheapest to most realistic:
- Backtest: model slippage and delays, segment by regime, and use walk-forward analysis.
- Paper trading: real-time data with simulated execution and the full risk stack active. Bugs and edge cases surface here.
- Small live size: only after the earlier stages hold up, with small positions and monitoring for deviation from paper results.
Backtests are the starting point, paper trading is a closer test, and live trading is the final exam. Paper trading is still a simulation and does not capture everything live execution does.
The Honest Answer
If someone asks what win rate to expect, the honest answer is that a backtest cannot tell you, and a short paper-trading run is not statistically significant. Treat backtest numbers as an upper bound, not a forecast. Our own paper bot ended its simulated run net negative and has since been retired (see the case study); see why we retired it.
Related Reading
Our paper trading bot has been retired. These posts cover why, and the difference between paper and live trading.