What testing ten variations does to your results
Testing ten variations in an afternoon is the thing a no-code builder is genuinely good at. It is also, done carelessly, the fastest way to produce a number that means nothing — and that has a name in the literature.
The best of ten is not the best
If you test ten variations and keep the one with the highest return, that return is a biased estimate of what the variation will do next. You did not find the best strategy; you found the variation that best fitted the noise in this particular sample. The selection step is what introduces the bias, and it happens whether or not you intended to search.
This is data snooping, and it has been studied specifically on technical trading rules. Sullivan, Timmermann and White (1999) re-examined the performance of technical trading rules while accounting for the fact that many rules had been tried on the same data [1]. White's Reality Check gives the method for asking whether the best of many strategies genuinely beats a benchmark, rather than merely appearing to [2].
Harvey, Liu and Zhu (2016) make the consequence concrete in a related setting: once you account for how many candidates have been tried, the significance hurdle any single survivor must clear goes up, not down [3].
The practical translation is uncomfortable and short: ten tests make your best result less trustworthy than one test would have.
Why it feels like the opposite
Because testing more feels like working harder, and because the tool reports each test in isolation. The Strategy Tester shows you the result of variation seven. It has no idea that variations one through six existed, so it cannot discount the number it is showing you — and nothing in the interface will ever tell you that you have searched.
Curve-fitting also leaves no trace in the source. The code is clean, the costs are honest, the backtest is correct. Only the history of your own search distinguishes a discovered edge from a selected coincidence, and that history lives in your head.
The count is part of the result
So the number to write down is not just the return. It is:
"Best of N variations tested, on this instrument, over this date range."
Report the return without the N and you have published something unverifiable. Report it with the N and a reader can judge it. This is the same reproducibility point as stating your TradingView plan when you quote a backtest — see running and maintaining a strategy for why the plan changes the result too.
A protocol that survives ten tests
The answer is not to stop searching. Searching is how anything is found. The answer is to fix the rules of the search before it starts, because every decision you make after seeing a result is another selection step.
Decide these four things before the first test
| Decide in advance | Why it must be before |
|---|---|
| the metric you will judge on | choosing the metric after seeing results is picking the one that flatters |
| the date range you will test on | shortening a range because it "isn't representative" is fitting |
| the number of variations you will try | otherwise you stop when you are happy, which is never a neutral moment |
| a hold-out window you will not look at | it is only out-of-sample if you have not seen it |
The fourth is the one people skip, and it is the only one that produces evidence. Reserve a stretch of history — the most recent chunk is the honest choice — and do not run a single test on it until the search is finished. You get exactly one look. If the winner falls apart there, the search found noise, and that is worth knowing in an afternoon rather than in live trading.
Prefer a plateau to a peak
This one costs nothing and is the most informative thing you can do with ten results. Line the variations up in parameter order and look at the shape:
- a peak — one good result with bad results on either side — is almost certainly a fit. Nothing about markets says that a 14-period lookback works and 13 and 15 do not.
- a plateau — a band of neighbouring values that all work acceptably — is a property of the instrument. Pick the middle of the plateau, not the highest point on it.
The highest bar in a set of ten is the one most likely to be high by luck. Choosing its neighbour costs you a little reported performance and buys you the only robustness evidence available without a second dataset.
Change one thing at a time
Ten variations that each differ in three ways teach you nothing about which change mattered. Ten that differ in one dimension — a single threshold, a single period — produce the parameter surface described above. It is the same number of tests and a completely different amount of information.
What the platform will let you do
Testing many variations quickly runs into limits that are not about statistics. Each run is bounded by the execution-time allowance, 20 seconds for basic accounts and 40 for others, and a strategy may place at most 9,000 orders in a backtest before it stops adding trades [4]. On a short timeframe over a long range, a variation can hit that cap and report a flat curve that looks like a bad strategy rather than a truncated test — check the trade count against the cap before you rank it.
And your date range is bounded by your plan's bar allowance [4], so every variation in the search sees the same, plan-determined slice of history. That is fine, as long as it is the slice you report.
Then write down the count
Best of ten, on this instrument, over this range, judged on this metric, with a hold-out result of X. That sentence is worth more than the return on its own, and it is the thing you will want when you look at this strategy again in three months and cannot remember how hard you had to look for it.
Tactix AI on this workflow
AlfaTactix includes Tactix AI: use Tactix Studio to describe your idea in plain language and draft timeframes, signals, filters, and risk into the same six-step Strategy Builder form — or open Tactix Guide on any step when you only need a field explained. Review every value, then export MQL5 or Pine Script from Code Generator (form-first — not untested prompt-to-code).