Every quantitative researcher has admired a backtest that looks flawless. The equity curve climbs smoothly, the drawdowns are shallow, and the parameters seem perfectly chosen. That is precisely the problem: they were chosen. In-sample optimization rewards whatever fits the past, and the past contains far more noise than signal. At Bountify, where AI research agents propose hypotheses around the clock, the sheer volume of candidates makes this danger larger, so our defenses must be stricter.
Repeated Honest Exams
Walk-forward analysis is our answer. We fit a strategy on one window of history, evaluate it on the adjacent window it has never seen, then roll both windows forward and repeat. Each segment is an examination the strategy could not study for. A single out-of-sample split can be lucky. A long sequence of honest exams, passed under changing market conditions, is far harder to fake.
The mechanics matter as much as the idea. Data must be split chronologically, with no information from the future leaking into the fitting window. Signals are computed only from what was knowable at the time, and evaluation windows are embargoed from the tuning process entirely. Any shortcut here quietly converts an out-of-sample test back into an in-sample one, and the resulting confidence is counterfeit.
Parameter Stability as Evidence
We also study how performance behaves across the parameter space. A genuine effect tends to produce a plateau: results degrade gracefully as parameters move away from the optimum. A fragile one produces a spike: one precise setting earns, and every neighbor loses. A strategy that works under exactly one parameter set is, in our view, a coincidence that has been dressed for the occasion, and we decline the meeting.
Stability across time is the second axis. When we re-optimize in each walk-forward fold, we watch whether the chosen parameters wander erratically or settle within a narrow band. Wandering parameters suggest the optimizer is chasing noise from regime to regime. Stable parameters, paired with consistent out-of-sample behavior, suggest the strategy has captured something structural about how the market actually works.
Pre-Registered Promotion Criteria
Before validation begins, we write down what passing means: the out-of-sample thresholds, the stability requirements, the cost assumptions, the risk limits. These promotion criteria are fixed in advance and cannot be renegotiated after the results arrive. A strategy that clears them advances to paper trading; one that fails is archived with its evidence. Moving the goalposts after the fact is simply overfitting by committee.
The discipline is expensive. It kills most of what our agents produce, including ideas we found elegant. But capital deployed on an in-sample illusion is capital already lost; the market merely schedules the announcement. Our standard is therefore simple to state and hard to live by: a strategy either survives repeated out-of-sample examination under rules written in advance, or it does not trade. Walk forward, or walk away.