Backtesting Mistakes That Only Real Market Data Reveals
The backtesting mistakes that survive a code review are data mistakes: invented rows, the wrong adjustment basis, and fields that vanish between runs.
The backtesting mistakes that survived our code reviews were never logic errors — they were defects in the price series the logic ran on. A data source that invented 117 sessions before an instrument existed, a series adjusted for splits but not dividends, a fill convention worth more than the strategy’s edge, and a filter field that returned nothing on one run and a real number the next: every one of those produced a clean-looking result that was wrong before a single line of strategy code executed.
This page is about how those defects were caught, and what it costs to catch them. It contains no positions, no signals and no account values.
The historical figures below come from our own research runs and are backtest simulations over stated periods — not live results, not predictions, and not advice.
Mistake 1: assuming every row your data source returns describes a day that happened
The most expensive defect we have recorded was a data provider that back-pads history for instruments that did not yet exist. From the migration record, verbatim:
hood back-pads pre-inception history with a flat constant (QLD had 117 fake rows at [REDACTED] before its 2006-06-21 inception). Trimmed in the cache build; QQQ/SPY unaffected.
One hundred and seventeen sessions of a single repeated price, sitting in front of a real series, with no error, no null and no flag. The API answered every request successfully. Nothing about the response said “this instrument did not trade on these dates.”
A repeated constant has zero return variance, so any risk-adjusted metric — Sharpe, volatility, drawdown, anything with a denominator built from dispersion — is computed partly over a stretch where the instrument by construction never moved. The result is not noisy. It is systematically flattering, in exactly the direction that makes a strategy look publishable.
The check is cheap and almost nobody runs it: before testing a rule, compare each series’ first bar against the instrument’s actual inception, and look for runs of identical closes. Two queries would have caught this instantly.
Mistake 2: not knowing which adjustment basis your price series uses
The same migration produced a second, quieter finding, recorded verbatim:
Faithful to 3 decimals — QLD/QQQ dividends are negligible, so hood’s split-adjusted series ≈ Yahoo’s split+dividend-adjusted series. (For SPY-heavy total-return work, note hood daily is split-only; document where a buy-hold benchmark appears.)
This is the shape of a defect that hides until it matters. On a low-yield instrument, a split-only series and a split-plus-dividend series agree to three decimal places — which means every test you run on those names validates cleanly and teaches you that the question is a technicality. It is not. The two series diverge in proportion to yield, so the same pipeline that was provably correct on a growth ETF is quietly wrong the moment a dividend-paying benchmark enters the comparison.
The trap is specifically the buy-and-hold benchmark, because that is where the omitted dividends compound uninterrupted for the whole test window. A strategy that beats a split-only buy-and-hold line may lose to the total-return one, and the chart looks identical either way.
“Which adjustment does my source apply?” is a question with a measurable answer. Ask it once per data source, write the answer down next to the cache, and note explicitly wherever a benchmark appears. Options data raises the same question at corporate actions: our options backtest split-handling write-up shows an unadjusted reverse split faking a +78% week.
Mistake 3: treating the fill assumption as a rounding detail
Most backtesting material disposes of execution in one line — “add slippage and commissions” — and that is adequate for a penny-wide equity spread. On options it is not a simplification; it is the variable under test.
On six years of real ThetaData NBBO quotes (2020–2026), one QQQ put-credit-spread configuration at 30–45 days to expiry and 0.16 delta returned a profit factor of 1.18 priced at the mid and 0.80 priced at the bid and ask, across 167 and 166 trades respectively. Same entries, same exits, same sizing, opposite verdicts. The win rate barely moved — 83% against 79% — which is the reason win rate is such a poor summary of a premium-selling strategy.
The correction most people reach for next is also wrong. Running the identical comparison on SPY moved profit factor from 0.76 to 0.79 over 292 trades: no meaningful drag at all, pointing the opposite way. The cost of the fill assumption is a property of the instrument, so a fixed slippage haircut applied across a portfolio of backtests is too large in some places, irrelevant in others, and looks equally rigorous in both. We work through that result in full in options backtest fill assumptions.
Mistake 4: believing a metric you have computed once, from one source
The back-padding defect in Mistake 1 was not found by reading code. It was found because the data migration had an acceptance step that recomputed the entire pipeline against the previous source and compared it metric by metric. Here is that comparison as recorded — a 200-day moving-average rule, recomputed over a cache spanning 2006-01-03 to 2026-07-24:
| Metric | Prior source | New source |
|---|---|---|
| 200-day CAGR | 21.2% | 21.2% |
| 200-day Sharpe | 0.79 | 0.789 |
| 200-day max drawdown | −48.2% | −48.2% |
| 200-day trades | 115 | 115 |
| QLD buy-and-hold Sharpe / max drawdown | 0.73 / −84% | 0.73 / −84% |
And from the mean-reversion sleeve, the same test on an RSI(2) rule: out-of-sample expectancy of +0.933% per trade against +0.934%, on 25 trades and an 88% win rate in both columns, with a correlation to QLD of 0.361 in both. (A low correlation alone doesn’t make a sleeve worth blending in; see our trend-following diversifier backtest.)
These are historical simulations over the stated window — not live results, not a prediction, not advice, and not a claim about what either rule would do next.
The numbers themselves are the least interesting part. What matters is that agreement to the third decimal across two independently sourced price series is a real signal about pipeline correctness, and it is the only mechanism that surfaced the padded rows — the padding was inside one source, so no amount of staring at one source would have revealed it. Reproduce the whole pipeline on a second source, or accept that you are trusting one vendor’s parser.
Mistake 5: filtering on a field that is not always populated
Replicating a momentum screen day after day, we hit a failure mode we had not anticipated: names dropped by a market-capitalisation gate not because they failed the size test, but because the field came back empty.
One name was dropped from our 2026-08-27 run for a null market cap, returned a real figure of roughly $26.7bn when re-probed about 24 hours later, and ranked fourth in the 2026-08-28 run. Nothing about the company changed; only the vendor’s answer did. Across four consecutive runs the size gate never once failed a name on an actual size test — its only observed effect was silently deleting names whose metadata happened to be missing at the moment the screen ran.
A filter backed by an intermittently populated field is not a filter. It is a sampler correlated with your data provider’s cache state, and it is worse than a permanently missing field, because a permanently missing one fails loudly. A plain-English screening layer hides this one level deeper, since it never shows you which field it chose — see what plain-English filters cannot fix.
The same rig produced a gentler version of the problem. Re-reading the same completed session’s volume three times over 48 hours, one name’s reported volume rose against an unchanged 20-session mean, moving its relative-volume reading 1.80 → 1.91 → 1.95. Closing prices were identical to the digit — the volume was revised after the close. On that pass it changed no rankings, which is precisely why it is worth recording: the mechanism is live whether or not it bit you, and a name sitting just under a threshold post-close can clear it by morning on revised volume alone.
Both findings collapse into one rule that most backtest archives violate: reproducing a historical screen requires the run timestamp, not just the bar date. Same bar, same code, different answer. The same requirement governs live workflows, where the timestamp and filter set have to travel with the shortlist — see how the screener, chart and journal hand off to each other.
Mistake 6: letting a filename be the evidence that a run happened
This one is ours, not a vendor’s. An automated probe on our own rig named its output file by UTC date while its schedule ran on Eastern time, and opened that file with a truncating write — so on 2026-08-26 an evening run and the next morning’s scheduled run resolved to the same path and the first was silently overwritten. We recovered it from version control, but the directory itself showed one file where two runs had happened.
Carry the lesson into any result archive: a correctly named file is not evidence that the run inside it is the run you think it is. Verify by embedded run timestamp. A silent overwrite does not only destroy data — it invalidates the monitoring that was supposed to notice. That failure came out of an automated rig rather than a research notebook, and the rest of what it taught us is in running an autonomous trading agent.
The checklist
- Compare every series’ first bar against the instrument’s real inception date, and scan for runs of identical closes.
- Record which adjustment basis each source applies, and flag every place a buy-and-hold benchmark is computed.
- State the fill convention explicitly, and run the test under both a generous and a pessimistic one.
- Recompute the whole pipeline against a second, independently sourced series before believing any headline metric.
- Treat every upstream field as optionally empty; log what a filter removed and why, not just what survived.
- Archive the run timestamp alongside the bar date, and verify runs by embedded timestamp rather than by filename.
Frequently asked questions
What is the most common backtesting mistake?
In our own records, trusting the price series. Overfitting gets the attention because it is the interesting statistical problem, but every defect above produced a clean, plausible, entirely wrong result without any curve-fitting at all — and none of them would have been caught by a code review, because the code was correct.
How do you know if your historical data is bad?
You compare it to something. A second source, an instrument’s known inception date, a vendor’s stated adjustment methodology, or the same figure recomputed a day later. A single source cannot be checked against itself, which is why our validation sequence puts data checks before rule checks.
Does a longer backtest fix these problems?
No — it enlarges them. A longer window means more history from a source you have not audited, and pre-inception padding lives specifically in the oldest part of a series, which is the part a longer test adds.
Should a backtest ever be re-run on the same data?
Yes, and it is one of the cheapest tests available. Two of the findings above — the revised volume and the intermittent metadata field — only appeared because a completed session was read more than once. If a re-run of an unchanged historical period produces a different answer, that is a data finding, not a nuisance.
This is educational content about research methodology, not financial advice. Every figure above is a historical backtest statistic from our own research over a stated window — not live results, not a prediction, and not a recommendation. No positions, signals or account values appear on this page. The same standard applies to our own published results, which is why we publish the trial protocol and the regime read method alongside them.