How do I tell if an ML paper's results are reproducible — and what should its artifact actually contain?

November 6, 2026 · 5 min read

SentX SI

The short answer: an artifact is reproducible when it contains the procedure, not just the code. A released repository with a README is the floor, and the floor is not enough. Anyone arriving at that repo cold — a new collaborator, a reviewer, or you — has to reconstruct everything the authors knew but never wrote down, and the quality of the reconstruction decides whether the paper's numbers come back. A paper submitted to arXiv on 5 November, ReproBridge (arXiv:2611.08452, by Yubo Wang, Rui Meng, Jaehyun Nam, Tomas Pfister, and Jinsung Yoon), puts numbers on exactly that gap, and its evidence gives you a way to triage any artifact before you invest a week in it.

Source: ReproBridge: Building Reproducibility into ML Research Artifacts

What ReproBridge measured

The paper's core observation is that reproducing an ML experiment means reconstructing procedures from information scattered across the paper, the released code, and supporting artifacts. For coding agents, it says, success depends on their ability to fill in missing details rather than on the released artifacts alone. That is a fair description of a human replicator too.

Source: ReproBridge: Building Reproducibility into ML Research Artifacts

ReproBridge-Bench tests the gap directly. It provides 1,029 tasks from 387 experiments across 36 papers, each task anchored by five human-verified reference scores. Agents attempt the tasks in fresh sessions — no memory of prior runs — either from the raw released code or from a refined artifact, and a separate Auditor agent checks the execution evidence rather than taking the reproducing agent's word. Across four evaluated models, which the abstract does not name, refined artifacts raised mean reproduction success from 76.4% to 92.6% and reduced downstream API spending by 19.7% on average. The spread between the best and worst model narrowed from 21.6 to 3.9 percentage points, with all four clearing 90%.

Source: ReproBridge: Building Reproducibility into ML Research Artifacts

Two caveats travel with those numbers. They are means across the bench, not a promise about any individual paper. And the fresh-session design is what makes them useful to you: a cold agent in a fresh session is structurally the same position as a reader opening a stranger's repository for the first time.

Source: ReproBridge: Building Reproducibility into ML Research Artifacts

The four components, in order of weight

The refined artifact the paper defines has four parts: a refined codebase, an environment setup, experiment launchers, and a reproduction guide. The ablation is the part that should drive your decisions. Removing any one component lowered success, and the largest losses came from removing the code and the environment. So the guide is not what carries the weight — runnable code and a pinned environment are. A beautifully written reproduction guide over a codebase that will not run, or a codebase whose dependencies are unpinned, fails for the same reason raw code fails: the procedure is still missing, just relocated.

Source: ReproBridge: Building Reproducibility into ML Research Artifacts

A triage checklist before you invest

The paper does not hand readers a checklist; it gives the design and the evidence for why each component matters. Organized around that, here is a triage I would apply before committing to a replication — my own reading practice, not a prescription from the paper.

Source: ReproBridge: Building Reproducibility into ML Research Artifacts

First, the code. Clone it and try to run something small. Does it die on imports, hardcoded paths, or missing config? The ablation says this is where most of the value lives, so treat a codebase that needs fixing before it runs as a raw-code artifact, whatever the README claims.

Source: ReproBridge: Building Reproducibility into ML Research Artifacts

Second, the environment. Are dependencies pinned — versions, not just names? A lockfile, a container image, or an exact requirements file. "Works on Python 3" is not an environment.

Third, the launchers. Can you map each experiment in the paper to a command? If the headline result comes from a notebook cell or a shell history, you are writing the launcher yourself.

Fourth, the guide. Does it state the procedure the paper leaves implicit — the ordering of steps, the decisions, the hyperparameters where the paper's table and the default config disagree? The ablation puts code and environment at the top of the cost ranking, so this is the check that decides your verdict only once the first three pass.

Source: ReproBridge: Building Reproducibility into ML Research Artifacts

Score it honestly. An artifact that passes all four is one where the remaining work is verification, not reconstruction. One that fails on code or environment is a reconstruction project, and you should budget it as such — the missing procedure is your work, not the authors'.

Source: ReproBridge: Building Reproducibility into ML Research Artifacts

A worked illustration

Here is an illustration of the triage, not a real paper I inspected. Suppose a paper reports a fine-tuned model beating a baseline by a few points, and the authors release a repository. You clone it. The README says "see configs/ for hyperparameters," but configs/ holds one file with a different seed and batch size than the paper's table, and nothing says which setting produced which number. There is no requirements file, just an import list you would reverse-engineer. The training command for the main result appears nowhere. On the checklist: code, weak — it runs, but the defaults do not match the paper; environment, fail; launchers, fail; guide, absent. Verdict: raw-code artifact. The science may be sound, but closing the gap is your project, and the bench's raw-code mean of 76.4% is the number I would price it against. Now imagine the same paper shipping pinned dependencies, one launcher per experiment, and a guide that says the table-2 result uses config B, seed 42, three epochs. Same science, different artifact — and a replication that is verification instead of archaeology.

Source: ReproBridge: Building Reproducibility into ML Research Artifacts

Checking the artifact the way you check a citation

The habit generalizes. SentX's model card for Victoria — a first-party disclosure, not an independent certification — states its verification advice in one line: open the cited source and check the supporting passage. Applied to an artifact, the same move is open the repository and check the supporting code. The card is equally plain that answers can contain incorrect facts, citations, or calculations, which is why the checklist stops at triage: it tells you whether the artifact is worth running, not whether the numbers inside it are right. One practical note if you use an assistant to read the paper alongside you: SentX's privacy policy states that submissions are not confidential and may enter training or shared memory, so a proprietary manuscript or an unpublished preprint is not material to paste.

Source: Victoria AI Model Card; SentX Privacy Policy

Reproducibility is not a property of a paper; it is a property of the distance between the paper and its artifact. Nothing short of running the code measures that distance exactly, but the four checks tell you which side of the gap you are standing on before you spend the week.

Source: ReproBridge: Building Reproducibility into ML Research Artifacts