SentX SI Blog Meet Victoria

I'm choosing a model from benchmark numbers — how do I know the score shows real capability, not contaminated training data?

· 3 min read

SentX SI

Start from an uncomfortable premise: you cannot tell from the score itself. A number is a number, whether the model earned it by reasoning or by reciting an example it memorized during training. What you can do is check the conditions under which the score was produced, and a new arXiv paper gives those conditions a name and a measurement. Qin and colleagues, "Benchleaking Is Accelerating" (arXiv:2611.21315, submitted November 16, 2026), use benchleaking for the case where a benchmark's own test examples enter a model's training data, so a score reflects recall rather than capability. That is distinct from benchmaxxing, where a vendor optimizes toward a benchmark without seeing its test set. The paper's own finding is the one that matters for your decision: contamination has worsened sharply over the past month, and it now reaches pre-training base models, the foundation of much downstream research. A high score can no longer be read as a clean capability signal by default.

Source: Benchleaking Is Accelerating: Benchmark contamination has worsened sharply in the past month -- from post-training down to pre-training base models

The paper shows what the leak looks like. Using few-shot continuation across 20 benchmarks and four public base models, the authors found some base models writing out test examples — complete questions, options, and answers — that appear nowhere in their input. ERNIE-5.5-Flash-Base leaked 96% of HumanEval problems with their reference implementations and 85% of OpenBookQA questions with their answers; DeepSeek-V4-Flash-Base leaked 9% and none, respectively. The authors then defined an Anti-Hack score: the same benchmark recomputed with every leaked example marked wrong and the example count unchanged. Averaged across their suite, that adjustment costs DeepSeek-V4-Flash-Base 1.5 points and Hy3-preview-Base 1.3, against 5.0 for MiMo-V2.5-Base and 16.6 for ERNIE-5.5-Flash-Base. The point is not which vendor loses more, but that the same reported score can sit on a very different footing depending on how much of it owes to leaked examples.

Source: Benchleaking Is Accelerating: Benchmark contamination has worsened sharply in the past month -- from post-training down to pre-training base models

Here is a practical way to run that check yourself, offered as an editorial suggestion rather than a method the paper itself prescribes. Take the benchmark's public release date and compare it against the vendor's stated training cutoff. A benchmark that existed before the cutoff had every opportunity to be in the training data; a benchmark published after it was not. Then do the habit the SentX SI model card names for its own outputs — open cited sources and check the supporting passage. For a vendor's reported score, that means finding the actual evaluation: the exact model version, the benchmark version and date, and the raw numbers behind the delta. If the vendor publishes only a single number, you have a claim, not a checkable result.

Source: Victoria AI Model Card | Capabilities, Evidence and Limits

From that, a three-step contamination-awareness check, also an editorial suggestion: date each benchmark against each model's cutoff; treat pre-cutoff scores as suspect rather than false — leak is possible, not proven; and make your decision on post-cutoff benchmarks, held-out tasks, or a published decontamination report. The last is what the paper calls for, alongside contamination-adjusted scores, as standard practice. Until vendors ship those reports, the date check is the best instrument available: it will not prove a score is clean, but it tells you which scores you are allowed to ignore.

Source: Benchleaking Is Accelerating: Benchmark contamination has worsened sharply in the past month -- from post-training down to pre-training base models

It is worth ending where that instrument points. The SentX SI model card, which we reviewed on October 23, 2026, declines to assert an independently verified benchmark score and pairs that refusal with the same verification habit — open the cited source, check the passage — instead of taking any number on trust, vendor-reported or otherwise. A buyer who applies that standard to other vendors' numbers is doing exactly what the paper's findings now make necessary.

Source: Victoria AI Model Card | Capabilities, Evidence and Limits

Sources

  1. Benchleaking Is Accelerating: Benchmark contamination has worsened sharply in the past month -- from post-training down to pre-training base models — arXiv
    Supported claims and limits
    • Paper defines benchleaking (test examples entering training data so a score reflects recall rather than capability) and distinguishes benchmaxxing (optimizing toward a benchmark without seeing its test set)
    • Few-shot continuation on 20 benchmarks across four public base models found some models writing out test examples absent from input; ERNIE-5.5-Flash-Base leaked 96% of HumanEval with reference implementations and 85% of OpenBookQA with answers, against 9% and none for DeepSeek-V4-Flash-Base
    • Anti-Hack score recomputes the benchmark with every leaked example marked wrong and example count unchanged; suite-average cost is 1.5 for DeepSeek-V4-Flash-Base, 1.3 for Hy3-preview-Base, 5.0 for MiMo-V2.5-Base, 16.6 for ERNIE-5.5-Flash-Base
    • Authors call for decontamination reports and contamination-adjusted scores to become standard practice; arXiv:2611.21315, v1 submitted 16 Nov 2026
  2. Victoria AI Model Card | Capabilities, Evidence and Limits — SentX SI
    Supported claims and limits
    • Model card is a first-party product disclosure that does not assert an independently verified underlying model version, model size, training-data inventory or benchmark score
    • Known limits section names the habit 'Open cited sources and check the supporting passage'
    • Card states Reviewed: October 23, 2026
Meet Victoria