Early in my quantitative trading journey, I did something that every new quant trader does: I built a strategy that looked amazing on paper and absolutely died in live trading. It had a 92% win rate in backtest. It had a 5.8 profit factor. It never had a down month. I thought I’d cracked the code.
Three months later, I shut it down after a 17% drawdown.
This is the story of how I fell for the classic over-optimization trap, what I learned from it, and the framework I now use to test strategies before I risk real money on them.
The Strategy That Looked Too Good to Be True
I was testing a mean-reversion strategy on EURUSD 1-hour charts. The basic idea was simple: when RSI goes below 20 and price is near the lower Bollinger Band, go long, and exit when it bounces back to the middle band.
But then I started “optimizing.” I changed the RSI period from 14 to 9. I changed the Bollinger Bands from 20/2 to 17/1.8. I added a MACD filter. I added a time-of-day filter (only trade between 4 a.m. and 10 a.m. GMT). I added a volatility filter using ATR.
Each little tweak made the backtest look better. Win rate went from 58% to 67% to 76% to 84%. Profit factor climbed from 1.4 to 2.3 to 3.8 to 5.8. I was giddy.
Here’s what the final backtest showed on 3 years of EURUSD 1-hour data:
| Metric | “Optimized” Strategy |
|---|---|
| Total Return | +247% |
| Win Rate | 91.7% |
| Profit Factor | 5.8 |
| Max Drawdown | -2.3% |
| Number of Trades | 184 |
247% return in 3 years with only a 2.3% drawdown? If this was real, I’d be retired in a few years. I knew deep down it was too good to be true, but I wanted to believe it.
Live Trading: Reality Hits
I funded a small live account with $2,000 and started trading the strategy manually. First week: 3 wins, 1 loss. Great — “it works!” Second week: 2 wins, 2 losses. Hmm, lower win rate but still profitable. Third week: 0 wins, 4 losses. Now I’m down 8% and confused.
By week 12, I was down 17%. The strategy was losing more often than it was winning. The average loss was bigger than the average win. Everything that worked in backtest was failing in live trading.
What happened? I had curve-fit the strategy to historical data. All those little tweaks I made — the 9-period RSI, the 17-period Bollinger Bands with 1.8 standard deviations, the exact time window — they didn’t find a real market pattern. They just found the exact combination of parameters that would have worked perfectly on that specific 3-year dataset.
When market conditions changed even slightly, the strategy fell apart because it was never based on a real market edge. It was based on noise.
The Warning Signs I Ignored
Looking back, there were so many red flags. I just chose not to see them because I wanted the strategy to work:
1. Too many parameters. A strategy with 6-7 tunable parameters (RSI period, Bollinger period, Bollinger deviation, MACD settings, time window, ATR filter period) gives you thousands of possible combinations. With enough combinations, you can always find one that fits historical data perfectly.
2. Perfect equity curve. The backtest equity curve was almost a straight line up. Real trading strategies don’t have straight-up equity curves. They have drawdowns, flat periods, rough patches. A perfect backtest is a warning sign, not a goal.
3. No out-of-sample testing. I tested the strategy on the same data I used to build it. Of course it looked good — I built it to look good on that data. A proper backtest requires setting aside data you don’t look at until the strategy is finished.
4. Tiny average profit per trade. The strategy only made an average of 4 pips per trade after spread. That means any slight change in market conditions, spread, or execution costs would erase the edge entirely.
How I Test Strategies Now
After that expensive lesson, I rebuilt my entire strategy testing framework. Here’s the process I use now:
Step 1: Concept first, parameters second. Before I touch any numbers, I write down why the strategy should work. What market inefficiency does it exploit? Why would it persist? If I can’t explain it in plain English, it’s not a strategy — it’s a curve-fit.
Step 2: Simple rules, few parameters. Good strategies have 2-3 parameters max. The more knobs you have to turn, the more likely you are to over-fit. If a strategy needs 6 parameters to work, it doesn’t work.
Step 3: In-sample / out-of-sample split. I take my historical data and split it. I build the strategy using only the first 70% of the data (in-sample). Then I test it on the remaining 30% (out-of-sample) and compare results. If the performance drops by more than 30-40%, it’s probably over-fit.
Step 4: Walk-forward analysis. Instead of a single backtest, I run the strategy through multiple rolling windows. Optimize on one period, test on the next. This simulates what it would actually be like to trade the strategy in real time, making decisions with only past data.
Step 5: Stress testing. I test the strategy on different pairs, different timeframes, and different market regimes. A strategy that only works on EURUSD 1-hour from 2019-2021 isn’t a strategy — it’s a historical anomaly.
The Right Way to Think About Backtesting
Here’s the mindset shift that changed everything for me: a backtest isn’t there to prove a strategy works. It’s there to try to prove it doesn’t work.
Your job as a strategy developer isn’t to make the backtest look good. It’s to attack the strategy from every angle — different parameters, different markets, different time periods — and see if it still holds up. If you can’t break it, maybe it’s real.
If you’ve ever built a strategy that looked amazing in backtest and terrible in live trading, you’re not alone. It’s basically a rite of passage for quant traders. The key is to learn from it, build a better testing framework, and never again fall in love with a perfect equity curve.
The market always finds a way to humble strategies that are too perfect.
Get more strategy development insights: Join Telegram @DongyiTrade or email contact@dongyitrade.com.
