A self-audit of my own trading strategy. 256 parameter variants. Zero deployed.
A validation pipeline built to falsify a strategy rather than confirm it. I ran it against the directional momentum entry rule from my own Hyperliquid perpetuals bot, across four symbols and four market regimes. Every one was rejected. Nothing went live.
This repository is the record of that.
Backtests show profit, live trading loses.
Search a grid of a few dozen parameter combinations and one of them will always produce a pretty equity curve. The question is whether that is an edge or the best-fitting noise. Without correcting for the search itself, the two are indistinguishable — and most published backtests never make that correction.
Four gates. A candidate must pass all of them.
| Gate | Threshold | Why |
|---|---|---|
| Out-of-sample sample size | ≥ 100 trades | below this a t-statistic means nothing |
| Edge after fees | > 0 | round-trip fee floor 0.0258% (maker+maker) |
| Statistical significance | t ≥ 2.0 on the test set | not on the set the search optimised over |
| Parameter-neighbourhood survival | ≥ 60% | a real edge is a plateau, not a spike |
Plus a Bonferroni correction on the training result, because the argmax of a 64-combination search produces false winners on pure noise.
Two design choices that changed the outcome:
- Barriers are evaluated against real candle high/low, not a simulated wick. An earlier version used a simulated intrabar range roughly 21× too narrow for a 4h candle, and that alone flipped BTC's out-of-sample edge from clearly positive to negative. The apparent edge was an artefact of the simplification.
- The neighbourhood test is the decisive gate. If a parameter set works and its immediate neighbours in the grid do not, it is curve-fitting. Three of four symbols failed here.
Directional momentum entry. Binance archive data, real OHLC, 50/50 train/test split.
| Symbol | TF | Candles | Period | OOS trades | Edge OOS | t | p OOS | Neighbourhood | Verdict |
|---|---|---|---|---|---|---|---|---|---|
| BTCUSDT | 4h | 10,950 | 2021–2026 | 136 | −0.0923% | −0.50 | 0.615 | 58% | REJECT |
| ETHUSDT | 2h | 13,140 | 2023–2026 | 908 | +0.0065% | +0.10 | 0.923 | 23% | REJECT |
| LINKUSDT | 2h | 13,140 | 2023–2026 | 1,039 | −0.0790% | −1.26 | 0.208 | 0% | REJECT |
| SOLUSDT | 4h | 4,380 | 2024–2026 | 230 | +0.3137% | +1.59 | 0.111 | 93% | REJECT |
0 of 256 combinations passed all gates.
Two symbols posted a positive out-of-sample edge. Neither survived. SOL looked closest — +0.31% per trade with 93% neighbourhood survival — and still failed on significance: t = 1.59, p = 0.11, on the shortest history in the set, with the test window outperforming the training window. That pattern is a signature of noise, not of an edge that was hiding.
Every symbol's training-set p-value became 1.0000 after Bonferroni correction across the 64 combinations searched. In other words: nothing found in training was distinguishable from what the search would have found in random data.
An earlier sweep over 1,728 configurations on short intervals reached the same conclusion by a different route. Mean net result was negative for every momentum lookback window (−0.048% to −0.083% per trade), and the handful of "profitable" configurations turned out to be an artefact: identical trade counts and identical returns across different TP/SL settings, which means the barriers were never binding. That bug is exactly what the diagnostic stage of this pipeline now catches before validation runs.
Live trading agreed. 134 round trips over four days: direction accuracy 49%, gross P&L flat, and the entire net loss was fees.
- No live results. Nothing passed validation, so nothing was deployed.
- No cherry-picking. The best of 256 runs is not presented as evidence.
- Not a claim that the strategy is bad — a claim that the data cannot distinguish it from noise.
- Not investment advice.
Directional momentum is falsified for this bot: on 1m–15m, on 2h and 4h, on four symbols, in live trading, and under a fee-aware barrier model.
The obvious alternative — delta-neutral funding-rate arbitrage — was costed rather than assumed. Median funding on the venue sits at the interest-rate baseline, capping the path near 10.9% APR gross and roughly 6.5% net on a one-week hold. Switching exchanges does not raise the ceiling. Real, but too thin to build on at small size.
So: still looking. That is the honest state of it.
Python · NumPy · Hyperliquid Info API · Binance klines archive
I built this because I had a strategy I wanted to believe in. The framework said no, and that is the part worth showing — a tool that can reject its own author's idea is worth something.
I run this audit for traders who want the same answer about their own system before they put capital behind it.
mr73biio.github.io · ruszczak11@gmail.com
Pulsus Cordis Tui