Why a Single Backtest Can Lie
A great-looking backtest and a strategy that actually works are not the same thing. Here's how to tell them apart — with any tool, including ours.
Test enough combinations, and something will look brilliant
Flip a coin ten times and you'll probably get an unremarkable mix. Ask a thousand people to do it, and someone will get ten heads in a row. Nobody calls that person a coin-flipping genius — yet in backtesting, traders make exactly that mistake every day.
Every parameter optimization, in any tool, works the same way: test many combinations, surface the best ones. That's genuinely useful — but it carries a built-in catch. Some combinations will look great by pure chance, because random noise in historical data occasionally lines up with a specific setting and produces a beautiful equity curve that means nothing.
And the more combinations you test, the more likely it becomes that your "best" result is one of those lucky accidents. This isn't a flaw in any particular optimizer — it's a mathematical property of picking the best of many. It just means the top of any results table should be read with one question in mind: is this a real edge, or the lottery winner of this run?
Three signals worth checking before you trust any result
You don't need our tool for these. They're tool-agnostic, and they work in a spreadsheet.
How many trades is it built on?
A 300% return over 14 trades is an anecdote, not evidence. Small samples produce extreme outcomes in both directions — and extreme luck looks identical to extreme skill. The fewer the trades, the less a backtest can tell you.
Do the neighbors agree?
If period 14 makes money but 13 and 15 both lose it, you haven't found an edge — you've found a spike. A parameter value worth trusting usually sits inside a whole region of decent results, because a real edge doesn't vanish when a setting moves one notch. Isolated peaks are the classic signature of overfitting.
How hard did you search?
The best result out of 20 combinations and the best out of 2,000 are not equally meaningful. The wider the search, the more chances luck had to produce a winner — so the more impressive a result needs to be before it clears the "could just be chance" bar.
Checking all three by hand, for every run, is tedious. Which is why almost nobody does it. That's the actual problem we set out to fix.
How RunOpti reports these — automatically, on every run
RunOpti automates the parameter sweep — but the part we care about most is what happens after the backtests finish. Every result is scored against the three signals above:
- check_circle
Low-sample flag. Results built on fewer trades than a minimum threshold are explicitly marked low-confidence — no matter how spectacular the raw numbers look.
- check_circle
Neighborhood stability. Each combination's score reflects how consistently its neighboring parameter values perform. A lone spike scores lower than a solid region, even when the spike's raw return is higher.
- check_circle
Deflation for search size. Scores are adjusted for how many combinations the run tested. A best-of-2,000 has to clear a higher bar than a best-of-50, because it had forty times as many chances to get lucky.
- check_circle
And when nothing passes, we say so. If no parameter region survives these checks, RunOpti tells you directly: no reliable region found. We think a tool that can't say "this run produced nothing trustworthy" isn't really measuring trustworthiness at all.
That last one is the feature you'll like least in the moment — and the one that makes the rest of the report worth reading.
A note on what these scores mean. RunOpti's robustness scoring is a relative ranking within a single run — it identifies which of the tested combinations are most trustworthy relative to each other. It is not a guarantee of future performance. Past results — however carefully validated — do not guarantee future returns.
See what your strategy looks like under honest scoring
Run a full parameter sweep on your own strategy and see the whole picture — the robust regions, the lucky spikes, and the difference between them.
No pressure either way. If the honest answer is "no reliable region found," you'll want to hear it from your optimizer before you hear it from the market.