Articles ·
Backtest audit, part 4: checking the backtest itself
Part 1 fixed how the engine trades, part 2 what it is given and part 3 how its results read. Each fix came with tests written for it. This part adds two checks that do not look for any one mistake: they hold the whole engine to two properties, and they run on every change to it.
Nothing from the future
A backtest is honest only if, on every bar, the strategy knows what a live run would know by then and nothing more. A peek can come from anywhere: an indicator computed over the whole series, a range bar stamped earlier than it closed, a funding rate applied before it was published, another market's bar read before it closed, a setting worked out from all the candles at once.
So the check does not inspect any of them. It replays a strategy, then replays it again with everything after a random moment replaced by a different future — the candles, the finer candles inside them, the other market's candles and the funding rates — and requires everything before that moment to stay exactly as it was:
- every decision, compared as the whole plan the strategy emitted on its bar: side, size, stop, target and the prices it saw;
- the account's lowest, highest and closing equity on every bar;
- every trade closed before the moment, field by field.
Decisions are compared, not only trades, for a reason. The classic look-ahead — deciding on a bar's close before that bar has closed — changes the decision taken just before the moment, and that decision trades only after it. A check that compared trades alone would never see it.
The check runs on all 32 indicators in the catalog, the five templates, a grid and a martingale, a strategy with managed exits and a session filter, and one that reads another market, in eight ways of asking for a replay: plain; warmed up on 100 earlier hours; with funding; at 20× under a three-step margin schedule; on range bars sized in percent and in price steps; on quarter-hour candles under hourly decisions; and under the earlier signal-close model. In all, 500 replays, with 83,751 decisions, 57,681 bars and 11,587 trades held equal before their cuts. The same input must also give the same result twice.
It found nothing: no decision in the engine can see anything after its bar closed. To be sure it would have seen one, we planted the smallest peek we could think of — every bar's volume replaced by the next bar's — and ran it. It failed on Relative Volume, on the decision taken on the bar just before the cut.
A second simulator
The engine's simulator has grown with every rule the audit added: a bar walked both ways, gaps, resting orders, ladders, liquidation on the market's margin schedule, funding. Its tests check each rule on cases written for it; they cannot tell whether the rules still combine as described when everything happens at once.
So the rules are now written a second time, as plainly as they can be — one position, one resting order, one ladder, and on each leg of a bar a list of what the price can meet there — in a test that knows the engine only through its public calls. Then 4,000 random scenarios go through both. The accounts have fees from 0 to 10 bps, slippage up to 10 bps, leverage from 1× to 50×, a flat or a three-step margin schedule, and resting orders that expire. The markets have gaps, bars that do not move and long wicks, with funding settling at bar opens and inside bars. The decisions are random: market and post-only entries both ways, turns, managed exits, grids and martingales, sizes up to what the account can carry and beyond.
The two must agree to the last digit on every field of every trade, every bar's lowest, highest and closing equity, the drawdown and every count: signals, fills, expiries, rejections, skipped and refused entries.
They agree on all 4,000. The only differences on the way were in the 28th significant digit, where the second simulator added the same numbers in another order; it now adds them as the engine does, the wallet being the starting capital plus the net result.
As with the first check, we made sure this one can fail. Four mistakes planted in the engine, one at a time:
| Planted mistake | Caught on |
|---|---|
| A post-only buy priced exactly at the ask accepted instead of rejected | the 3rd scenario |
| A funding payment to the account kept on a bar that closed the position | the 3rd scenario |
| A tie between a bar's two ways resolved the other way | the 15th scenario |
| A ladder's remaining orders left resting after its position closed | the 3rd scenario |
What this proves and what it does not
The two checks show that the engine does what its rules say and never reads the future. They cannot show that the rules are the exchange's: two simulators written from one description agree with each other, not with Binance. That comparison is what the first three parts made, rule by rule, against the exchanges' own documentation, and it leaves approximations, each stated where it applies:
- stops and targets trigger on the bid or ask built half a spread from the bar's prices, not on the last trade or the mark price;
- liquidation is priced on the mid, where an exchange uses its mark price;
- a candle has no order book, so a market order fills at its touch plus the run's slippage, whatever its size.
How the two checks work, and how to run the first one on your own strategy: How the Metrand backtest is tested.