How do I know my agent's improved score is real — and not just the result of re-running the test until it passes?

November 3, 2026 · 6 min read

SentX SI

If the agent — or whoever set it up — can choose what gets tested, when testing stops, and which result gets reported, then an improved score is not evidence of improvement. It is the expected output of the procedure. A paper submitted to arXiv on November 2, 2026, "Can We Stop the Backtest Yet? Anytime-Valid Promotion Tests for Adaptive Financial Research Agents," by Eric Benhamou and five co-authors, and accepted at ICAIF 2026 in Milan, formalizes exactly this problem and proposes a fix.

Source: Can We Stop the Backtest Yet? Anytime-Valid Promotion Tests for Adaptive Financial Research Agents

The failure mode, in the paper's own setting

The paper's setting is financial research agents that can generate and monitor trading rules faster than a human can document the resulting selection process. Candidates appear at data-dependent times, weak rules are abandoned, and promising ones are watched until they cross a reporting threshold. That description generalizes further than finance: any agent that proposes variants, benchmarks them, discards the failures, and reports the best survivor is doing the same thing.

Source: Can We Stop the Backtest Yet? Anytime-Valid Promotion Tests for Adaptive Financial Research Agents

The problem is that the test is no longer fixed in advance. If you flip a fair coin until it lands heads, you will eventually get heads, and the chance of never getting heads in N flips is 2 to the minus N. A score threshold or a p-value is designed for one test on fixed data. Point it at a moving target — new candidates born when the data suggests, monitoring that continues until something crosses — and the threshold becomes a door that opens by construction. The paper's replay quantifies how fast: in a retrospective pseudo-out-of-sample replay with 24 panel-rule hypotheses, 3,000 shared-sign null paths, injected alternatives, and public ETF and industry data, repeated unadjusted tests produced a family crossing in 94.5% of the null paths. When nothing was actually true, the naive procedure found a winner almost every time.

Source: Can We Stop the Backtest Yet? Anytime-Valid Promotion Tests for Adaptive Financial Research Agents

What PROMOTE changes

PROMOTE restructures the test so that the act of watching is accounted for, in three moves. First, every candidate is assigned predictable testing wealth at birth: a fixed slice of a shared statistical budget, decided before any of that candidate's results are seen. Second, evidence is accumulated only from subsequent returns, through a cost-aware e-process — scores net of costs, measured only on data from after the candidate existed. Third, if the lifetime allocations sum to at most one, a weighted Ville argument bounds the probability of ever promoting a rule whose declared bounded net score has always had nonpositive conditional mean.

Source: Can We Stop the Backtest Yet? Anytime-Valid Promotion Tests for Adaptive Financial Research Agents

In plain terms, the e-process behaves like a betting bank account. Under the declared null, its expected value can only stay flat or shrink, so the chance that it ever grows large enough to trigger a promotion is capped by the size of the initial allocation. Because the cap is fixed at birth, watching longer, abandoning losers, and spawning new candidates on data-dependent triggers all cost nothing extra. The bound holds under dependent candidates, adaptive births, and indefinite monitoring. That is the "anytime-valid" part: you may stop whenever you like, including "when it passes."

Source: Can We Stop the Backtest Yet? Anytime-Valid Promotion Tests for Adaptive Financial Research Agents

The authors audit the implementation in two settings. In the replay above, PROMOTE showed 0.0% family crossing across the 3,000 null paths, against 94.5% for the unadjusted procedure; at a 15-basis-point aligned ETF-component perturbation it reached 70.1% target power with a median crossing at session 2,327 — it still finds real effects, just more slowly, on a panel of that size. In a separate exact mean-zero experiment where an agent selects candidate identities and birth times from past scores, the funded gate had 1.77% false-family crossings across 3,000 programs. And in the market replay, zero promotions after costs: on real data with costs included, nothing survived.

Source: Can We Stop the Backtest Yet? Anytime-Valid Promotion Tests for Adaptive Financial Research Agents

What a promotion certificate does and does not say

The paper states its own caveat, and it is the one to keep. The certificate controls historical false promotion under the declared null; it is not a certificate that an earlier edge remains profitable at deployment. A valid promotion test answers "did the procedure ever let a false candidate through?" — a statement about the testing process, made over its history. It does not answer "is this candidate good?" or "will it stay good?" The declared null is an assumption about the score distribution — bounded, with nonpositive conditional mean — and if the true process violates it, the bound is only as good as the declaration.

Source: Can We Stop the Backtest Yet? Anytime-Valid Promotion Tests for Adaptive Financial Research Agents

A checklist for reading adaptive self-evaluation reports

This is our editorial suggestion, not a finding of the paper. When an agent or a vendor reports an improved score from a self-run evaluation, the number is the least interesting part. The protocol is what to check.

1. Was the candidate set fixed before testing, or did candidates appear, get abandoned, or get extended based on results? If it was adaptive, an unadjusted threshold is the 94.5% scenario.Source: Can We Stop the Backtest Yet? Anytime-Valid Promotion Tests for Adaptive Financial Research Agents
2. Is there a declared null — a stated assumption that scores would not systematically improve under the baseline? Without one, "significantly better" has no denominator.
3. Are costs netted out of the score? A score that ignores the cost of running the test can be inflated by running more tests.
4. Is evidence accumulated only from data after the candidate existed? Counting pre-birth data as evidence breaks the bound.
5. Does the reported figure cover the whole family — every candidate ever tested — or only the survivor that got reported? A per-test figure on the best of many is the classic error.
6. How many candidates, over what horizon, on what data? The bound's strength depends on the allocation budget and the monitoring window, so a result scoped to a 24-hypothesis replay panel is a statement about that panel, not a universal guarantee.Source: Can We Stop the Backtest Yet? Anytime-Valid Promotion Tests for Adaptive Financial Research Agents

That last point matters for the paper's own figures: the 94.5% to 0.0% swing belongs to the 24-hypothesis replay, and the 1.77% to the 3,000-program mean-zero experiment. What transfers to your own evaluation setup is the method — budget at birth, evidence after birth, a family-level bound — not the specific percentages.

Source: Can We Stop the Backtest Yet? Anytime-Valid Promotion Tests for Adaptive Financial Research Agents

The habit, extended

SentX's public model card lists, under known limits: open cited sources and check the supporting passage. The card is a first-party product disclosure, not an independent certification — a label worth keeping attached to anything a company reports about itself. The same habit extends naturally to agent self-reported metrics: do not just check the number, open the protocol. What was the null, what was the budget, when did candidates die, and what did the tests cost? A score you can re-derive from a stated protocol is a measurement; a score that exists only as a headline is an anecdote.

Source: Victoria AI Model Card

One scope note: this is statistical methodology for adaptive evaluation, not financial advice. The trading framing is the paper's setting, and the lesson applies to any self-evaluating system.