Fourteen studies validated a hindsight stop

Draft 1 — 2026-08-29. Every figure below is from a committed research record: sealed pre-registrations, result files, and broker exports. Nothing is remembered.

For most of 2026 I ran an automated futures strategy on two prop-firm evaluation accounts from a VPS — a rented, always-on server. The six-year backtest behind it showed +1,663R on NQ at a profit factor of 3.03, and +923R on YM at 2.06 — roughly $1.1 million on a $50k account with a 2.8% maximum drawdown. (R is profit measured in multiples of the risk each trade's stop-loss defined; profit factor is gross wins divided by gross losses.) It had survived a train/holdout split — tuned on one period, judged on a later period never touched during tuning — and walk-forward testing across thirteen half-year windows, which re-checks the edge window by window through time, a look-ahead audit that found and fixed a real bug, and fourteen separate parameter studies, each with a control arm, each reproducible from a command line I had written down. The test suite, by the end, held 815 tests.

The live result, reconciled against the broker's own export, was 32 positions and a net of $55. The configuration actually deployed at the end had nine positions, one winner, and −$1,280.

This is an account of why the backtest was wrong, why fourteen studies agreed with it, and what I now think a trader should measure before buying an evaluation. The strategy itself is not the point. I'll describe it in one paragraph and then not defend it.

The strategy, in one paragraph

An ICT-style model (the popular retail methodology named for its creator, the "Inner Circle Trader"): a liquidity sweep on the 1-hour chart, a market-structure shift on the 5-minute, an entry at a fair-value gap or order block, a stop under the 5-minute pivot, a target at the next draw on liquidity. Long and short, NY session, one instrument per account, 0.5% risk. If you trade futures intraday you have seen a hundred variants of it. Mine was at least honestly built: causal bias, no repainting, a live engine that imported the same detection functions the backtester used so the two could not diverge.

That last part turned out to matter, in the wrong direction.

The bug

The simulator started checking the stop on the bar after the fill. One line, commented "avoids instant stop-outs". Whoever wrote that comment — it was me — thought of it as a fidelity choice: a limit order that fills at the zone and then gets stopped on the same bar is noise, surely.

It is not noise. It is where the strategy's edge lived.

The stop was placed under the pivot of the 5-minute bar the entry fired on. The simulator knew that bar's full range when it placed the stop, because it placed the stop after the bar had closed — and then it declined to test whether the bar's own wick crossed it. So the stop sat, structurally, just under the wick of a bar that had already happened. A stop anchored under the fill bar's own wick, exempt from the fill bar, is a stop that cannot be hit by the move that defines it. That is hindsight, wearing a plausible-looking rule.

When the fill bar was allowed to hit the stop:

YM went from +923R to +40R, profit factor 1.06, with 45% of trades stopped on their fill bar. Strip the eight largest winners from either honest book and both are negative (NQ −312R, YM −299R). A bootstrap — re-estimating the average by resampling the trades thousands of times — gave the honest NQ book a mean of +0.22R per trade with a 95% confidence interval of [−0.68, +1.55]. The right description of that book is indistinguishable from zero, not "+76R".

The ablation was the ugliest part. Turn off just the fill-bar check and the six-year figure comes back to +811R. Turn off just the live-style stop placement and it comes back to +1,608R. Every other simulator change — sweep confirmation at close, position caps, friction, killzone-only fills — moved the book by less than 10%. The edge was not a feature of the market. It was the pair: a tight stop computed with knowledge of the bar, and a simulator that skipped that bar.

On the last two months of the book, the honest run and the no-fill-bar-check run share seven trades, six of them stopped on the fill bar. Turning the check off flips the slice from −7.6R to +140.4R — and 136R of the recovery is a single trade whose stop was four ticks wide. That is what the legacy edge was made of: stops the fill bar should have taken, and a lottery ticket when it didn't.

Why fourteen studies didn't catch it

This is the part I think is worth more than the bug.

Every study I ran compared an arm to a control on the same simulator. A walk-forward test asks whether the arm beats the control window by window. A train/holdout split asks whether the arm's edge persists out of sample. A look-ahead audit asks whether the signal uses information from the future. None of them asks whether the fill model is honest, because both arms share it, so an artefact in the fill model is present in both sides of every comparison and cancels.

When I finally ran the honest simulator against the legacy one as if they were arms, the honest one was worse in 12 of 13 windows on NQ and 12 of 14 on YM, t ≈ −4.9. The artefact was uniform in time. That is precisely why walk-forward had passed it: walk- forward validates that an effect is stable, and this one was perfectly stable, because it was arithmetic.

Addendum (2026-08-31). After this was published, a reader on r/algotrading generalised the point better than I had: any quantity the simulator can see at decision time that the live system couldn't is invisible to every arm-vs-control comparison — and to walk-forward — because both arms read it. Mine arrived through the bar's range. It can just as well arrive through the dataset: a later revision of economic data or an adjusted continuous contract leaking backward into an old signal. Same door, different hinge. If your inputs can be revised, seal them point-in-time to what was published on the date of each decision. The same thread's other correction I have adopted: reconcile your fills against the same period's backtest weekly, not once at the end — mine reconciled money daily and mechanics only at the autopsy.

The individual studies were, in their own terms, good. A look-ahead in the bias calculation was found and fixed — the pre-fix numbers were re-derived rather than cited. A hardcoded tick size that had under-risked YM fourfold was found and fixed. A partial take-profit the live engine never sent was found modelled in the sim and removed. A "CE-anchored stop" arm that showed +56% on the holdout was rejected by walk-forward (7 of 13 windows, top three windows carrying 89% of the gain) — a single train/holdout split is not enough for a configuration decision; bucket by period and count windows. A "past-zone" filter that looked like a win when I dropped its rows post-hoc scored a loss when I re-ran the arm, because rejecting a trade frees the next sweep for a worse one. Sub-10-tick stops averaged +6.8R in train and −1.0R in holdout, carried 12–33% of the full-book R, and could not be filtered out without costing more than they earned.

Each of those is a real lesson. Each of them was learned inside a simulator whose stop could not lose on the fill bar. Rigor applied to the wrong layer produces confident wrong answers, and the confidence is the dangerous part.

How it was found

Not by me looking at results. By an AI assistant — Claude — that I pointed at a snapshot of the repository with instructions to attack the simulator's code, not admire its results. Any reader can reproduce that step for the price of a subscription. It listed the optimistic mechanics one by one: the sweep was timestamped by its open, so the simulator acted on a close that hadn't printed; the entry bar was exempt from the stop; the stop and target were computed on a different bar from the one the live engine actually fired on.

I implemented each as a flag defaulting on, kept a --legacy-sim switch that reproduces every old figure byte-for-byte (so the historical record stays checkable rather than quietly becoming unreproducible), and re-ran everything.

Then I did the check that should have come first. I took the live engine's own signals from the two months it had been trading and scored them under the honest fill model on the bars the engine had actually seen: NQ, 31 fills, 28 losses, 71% stopped on the fill bar; median stop 57 ticks against a median adverse excursion — how far price ran against the position within its entry bar — of 55 ticks in that regime. The stop was, in that market, about the size of the bar it sat inside.

And then the broker. Tradovate's Performance and Orders exports for both accounts, July 10 to August 28: 32 round trips (17 NQ, 15 YM)¹, net $55; the top three winners made $3,430 against −$6,115 of losers. On NQ, 8 of 17 positions were stopped within five minutes of filling, five within ninety seconds, three within zero to five seconds — the order filled already past its stop. Matching the honest model's per-trade verdicts to the broker's: model-says-loss → broker-loss, 20 of 20; fast-stop agreement 27 of 28 (the miss stopped at 5.9 minutes).

At that point the finding no longer rested on a simulator. The simulator had merely been the last of three witnesses to agree.

What an honest simulator found afterwards

Having a fill model that matched the broker was, I told myself, the real asset. So I wrote a protocol — pre-register the parameter space, the holdout, the number of studies allowed and the numeric pass/kill thresholds, in a file, hashed, before the first run; gates read in order by a reviewer; a trimmed bootstrap that removes only lottery tickets; a block bootstrap, which resamples runs of consecutive trades together because results come in streaks — and budgeted three spikes across different edge families before asking whether intraday index futures was the venue at all.

Two things in that table are worth more than the results. First, the 15m stop rule fixed the execution problem — 6.5% fill-bar stop-outs instead of 64% — and the edge still wasn't there. The execution failure was a property of my stop, not of the venue; the absence of edge was a property of the venue. Second, the harness resolves an edge of about +0.06R per trade at a thousand trades, and it produced a signed negative result on one family — a fill model that manufactures nulls cannot produce a signed loser. The nulls were nulls of the ideas.

The one positive number the entire programme found was NQ's overnight leg: +0.021 daily ATRs per night, about $3k a year per micro contract, with a worst night of −$2,001 and a worst twenty-night stretch of −$3,470 on roughly $2k of margin. That is the equity premium. The honest instrument for holding it is an index fund.

What the goal did

I want to be precise about the causal chain, because I think it is the common one.

The target was a large income from a small stake. A small stake means evaluation accounts. Evaluations have trailing drawdowns on unrealised P&L, daily loss limits, consistency rules and flat-by-close requirements, so they reward a high win rate and a shallow equity curve. That means tight stops. Tight stops on 5-minute NQ get hit inside the fill bar. A simulator that skipped the fill bar was therefore the one that "worked", and a $1.1M, 2.8%-drawdown backtest did not trigger disbelief because it was the number the goal needed.

The goal selected the evidence. The fourteen studies were rigorous about everything except the assumption the goal had already made.

I was paying roughly $750–850 a month for the privilege — data, bridge, two VPS boxes, and evaluation fees for accounts that, by the protocol I eventually wrote, could not be traded until a strategy had passed a forward test that no strategy passed. Everything was cancelled on 2026-08-29. The monthly burn is now zero and what remains is the record.

The fourteen studies were never fourteen tests of the edge. They were fourteen looks at the same assumption — and the assumption was the one thing the goal had placed beyond question.

If you are about to buy an evaluation

These are the measurements I would want to have made before any of the above, in roughly the order they would have hurt.

1. Read your broker's export, not your journal. Your journal is what you believe happened. The Performance export is what happened. On two accounts over seven weeks mine disagreed by a phantom −$475 (recorded in the next day's commit message) and a −$185 mis-booked fill (pinned since by a regression test), and my engine once narrated a full day of fills on a day the broker filled nothing.
2. How many of your stops were hit within five minutes of the fill? If it is a large fraction, your stop is inside the noise of the bar you enter on, and your backtest — if you have one — almost certainly doesn't test that bar. Mine was 47% on NQ, at the broker.
3. Is your stop narrower than the median bar of the session you trade? Measure the bar per session; the 09:30 bar and a London bar are different animals. A stop smaller than the bar is a hindsight stop unless you can say why it isn't.
4. Expectancy ex your top 5%, with a confidence interval. Sort your trades by R, drop the top 5% if they exceed 2R (a +1R winner on a 1R-target system is not a lottery ticket; a 12R trade on a 2-tick stop is), and bootstrap the rest in blocks of ten consecutive trades, in time order — blocks of the sequence, never of the sorted-by-R list. If the interval includes zero, that is your answer, however the full book looks. Every book I had was positive in full and negative trimmed.
5. Count the months. Not "is the holdout positive" — how many windows are positive, and what share of the total do the best three carry. My best arm ever, by holdout, had 89% of its edge in three windows.
6. Understand what walk-forward cannot see. It compares two arms on one simulator. A defect shared by both arms is invisible to it. The independent witness is the broker.
7. Pre-register. Write the parameter space, the holdout, the number of studies you are allowing yourself, and the numeric kill thresholds, before the first run. Fourteen studies is not rigor; it is fourteen looks.
8. Do not buy the evaluation until the expected payout per fee dollar, computed on your trimmed stats under that firm's actual rules, is above one. A funded account is leverage on an edge. It does not create one, and at a 7% win rate with rare 100R winners it multiplies exactly the profile the rules are designed to fail. And if you do not have enough trades for the interval in item 4 to mean anything, that is itself the answer to this item.

None of this is a strategy. It is the part of the work that the strategy-sellers, the firms and the brokers have no incentive to do for you, because every one of them is paid whether or not your edge exists.

¹ Consolidation rule, for anyone re-deriving from the raw Orders export: positions are flat-to-flat — a position opens at the first fill and closes when the instrument is flat again, so fills added while a position is open belong to it, with no time threshold. A time-window grouping instead (e.g. same side within 90 seconds) splits the 2026-07-27 04:24/04:26 NQ adds, 96 seconds apart, and reads 33 positions with 9 of 18 fast stops from the same 38 raw fills. Same tape, one grouping rule apart.

I am turning the harness this produced into a report you can run on your own broker export — the eight measurements above, in that order, with the full-book P&L last. It never connects to your account; the export you upload is parsed, then deleted, and only the derived numbers are kept. No affiliate links, no evaluation discount codes, and it will tell you no if the answer is no. The free calculator — which uploads nothing at all — is at https://honestbook.io.

← the calculator