Overfitting — a good backtest number
is not a good strategy
I tested six strategy ideas on four years of data. I threw all six away. Not because the returns were small — because I could not trust the numbers. Here are the five ways a backtest lies to you.
01Look at the neighbours
The most common way to fool yourself
Every strategy has numbers you have to pick. How long to hold. How many days to average. The usual approach is to try a range of values and keep the one that scored best.
But how do you know the best one is real? You look at the values right next to it.
Look at the red line. Move the setting by 3% and the score collapses, then recovers, then collapses again. The blue line moves 20% and barely bends. It has a clear middle and similar values on both sides.
Measured properly, the red line is four times more sensitive to the same relative change in the setting.
Here is the part that matters. When I nudge the red setting by one step, the number of trades changes by less than 5%. It buys in the same places and sells slightly later. If the score still swings that much, the swing was not produced by the strategy. It was produced by where a handful of large trades happened to get cut off.
With the blue line I can use the middle value with a clear conscience. One step either way would have given me almost the same thing. With the red line, no value is defensible.
A sharp peak is luck. A broad hill is skill. You cannot tell them apart by looking at the peak — only by looking next to it.
02If it only works recently, the market did that
The most convincing way to be wrong
The standard practice is to split the data. Build on the first part, confirm on the second. If the second part looks good, people say the strategy "passed validation".
I added one condition to a strategy and got this.
| Full 4 years | Last year (validation) | |
|---|---|---|
| Original | 100 | 100 |
| With the condition | 66 | 344 |
On the validation window it looks more than three times better, even though the full period got a third worse. If I had judged on validation alone, I would have shipped it.
The reason was deflating. That condition cuts the time my money spends in the market by about a quarter. And the last year happened to be a deep bear market.
Less exposure, less loss. Nothing to do with the strategy being good. In a falling market, every strategy that trades less looks brilliant.
So I now cut the four years into four separate one-year blocks and require all four to be positive. If only one year is spectacular, that year did the work.
03You are probably peeking at the future
A cheat that never appears in the code
I built a strategy that buys when a certain indicator is unusually low. I defined "unusually low" as the bottom 2%.
That definition hid a cheat. I calculated the bottom 2% across all four years at once. So a decision made in 2023 was using data from 2026. That is not something you can do in live trading.
I changed it so the threshold only uses data available up to that moment.
| How the threshold was set | Profit per trade |
|---|---|
| Using all four years (cheating) | 100 |
| Trailing 6 months only | 52 |
| Trailing 1 year only | 58 |
Half of the edge came from seeing the future. Nothing in the code said "look ahead". I had simply taken an average over the whole dataset.
The test is one question: "Could I have known this number at the time?" Ask it every time you use an average, a percentile, or any threshold derived from the data.
04Without a control, you have measured nothing
A positive number on its own proves nothing
I had a signal that fires often — close to thirty thousand times in four years. Trading it was a disaster. Stopping there, the conclusion is "useless signal". That conclusion would have been wrong.
I built a control that buys at completely random moments, the same number of times. Comparing raw price movement, before fees:
| How I bought | Profit per trade |
|---|---|
| On the signal | +0.040% |
| At random | +0.008% |
| Difference = what the signal is worth | +0.033% |
The signal carried real information — more than four times better than random. The problem was that the information is worth 0.033% while a round trip costs 0.11% in fees.
Real signal, but worth a quarter of what it costs to act on it.
Without the control I would have filed it as "useless". It was actually "real but too small", which is a completely different conclusion — it changes how I judge the next idea that looks like it.
05If win rate rises while profit falls, it is a trap
The most pleasant way to lose money
I added a rule that sells once the position reaches a certain level — the familiar "take what you can and get out".
| Win rate | Profit per trade | Holding time | |
|---|---|---|---|
| Original | 51% | 100 | 100 |
| With the exit rule | 73% | 34 | 22 |
Many more wins, one third of the money. The trade log feels much better. The account does worse.
The reason is in how this strategy earns. Line the trades up by profit and the one in the middle is roughly break-even. Almost all of the profit comes from a small number of large winners.
A rule that sells at a level removes exactly those winners. It keeps the small gains and deletes the big ones. Whatever you call it — a profit target, a trailing stop — anything that caps the upside does the same thing.
The decisive evidence: the looser I made the rule, meaning the less often it fired, the better the strategy did. As firings fell from 367 to 113 to 29, the score climbed steadily toward the score with no rule at all.
The rule was destroying value every single time it worked.
+The measuring tape can be wrong too
This one is my own bug rather than a strategy problem, but the consequences were large enough to keep.
I combined a strategy that had only 21 trades with my existing set, and the combined result improved noticeably. Twenty-one trades cannot do that. The bug was in the comparison code.
When overlaying two equity curves I was keeping only the dates where both had values. A strategy with few trades ends early. So four and a half months vanished from the comparison window — and those months happened to be bad ones. Nothing improved; a bad stretch was deleted.
I fixed it to fill days with no trades as zero return. The "improvement" shrank to less than a hundredth of what it had looked like.
06What I actually do now
A return figure on its own tells you nothing. This is the order I go in.
- Check the neighbours. Nudge the setting slightly — is the result similar? If it jumps, the number is luck.
- Split the period. Cut it into one-year blocks. Are all of them positive? If only one is, that year did the work.
- Check for future data. Every threshold you use — could you have known it at the time?
- Compare against random. Would buying at random have produced a similar number?
- Check profit per trade against fees. Not win rate. The more often you trade, the earlier this check belongs.
If an idea fails these five, I do not look at the rest of the numbers. Five of my six ideas died right here. The one that survived did so because it lost less, not because it earned more, and I still have not put it into live trading.
Overfitting is not picking a good number. It is picking a number without knowing why it is good. If you cannot explain it, it is not yours yet.
댓글 쓰기