My agent passes every static check and still fails in production — what does a green result actually establish?

October 18, 2026 · 6 min read

SentX SI

A green static result establishes one thing: the system behaved correctly under the exact representation, fold scheme, weighting and task specification the harness encoded. It does not establish that the score survives a change of wording, that the task resembles anything a real customer will send, or that the numbers were read the way they should be read. Production failures that survive an all-green pipeline often land in one of three places, and two papers submitted on 15 October 2026 plus one published observation record give each of them a measurable shape. The value of mapping them separately is that they demand different fixes. Treating all three as "your eval was too easy" sends you auditing the wrong layer.

What each green light actually covers

Unit tests establish that specified inputs produce specified outputs along specified paths. They say nothing about constraints nobody wrote down, which is precisely the territory where deliveries break. An eval-suite score establishes measured performance on the benchmark's own encoding of a task, and the encoding does the work. In a paper called Shortcut Sensitivity in Agent-Safety Evaluations, Hasan Demirkiran and Jens Ernstberger show the headline number moving purely through measurement choices: on AgentDojo-Traj trajectories, exact-tool lookup falls from F1 of .901 with mixed-domain folds to .495 under a domain holdout, and on AgentHarm-Traj the released TS-Guard falls from .902 under row weighting to .719 with equal interaction weighting, while balanced accuracy rises. Same system, same data, different bookkeeping. Safety-harness results carry the same exposure at the decision level: on a frozen 64-step cohort, rewriting an INFORMATION marker lowered blocking by 8.33 percentage points and opaque tool aliases raised it by 11.46, both effects surviving subtraction of a formatting placebo, with all unsafe calls still blocked — the movement was false positives against inherited labels. Judge agreement is the weakest green of the four. Two judges agreeing establishes that they agree; if the rubric encodes a shortcut, both inherit it, and agreement measures the shared blind spot, not its absence. That last point is my reading rather than a finding in either paper, but it follows directly from what the first two demonstrate.

Source: Shortcut Sensitivity in Agent-Safety Evaluations

Failure class one: the eval scored a shortcut

The causal evidence matters more than the correlational kind. Demirkiran and Ernstberger's paired interventions show benchmark representation changing a guard's actual decisions, not just its measured score, and the paper's three-level framework — label predictability, observational guard association, paired decision sensitivity — doubles as a pre-flight test. Before trusting a green safety result, hold the property fixed and perturb the surface: reword instructions, reorder steps, rename tools, swap distractor details. Systematic score movement means the suite is scoring a representation, and the recommended remedies are concrete: domain group holdouts instead of random folds, evaluation-unit metrics instead of row-weighted averages, and placebo-controlled paired edits so you can tell a real effect from formatting noise. Scope limits belong in the ledger: the work is accepted at the NeurIPS 2026 TAE and AIWILD workshops rather than the main conference, and the Haiku-sized effects did not reproduce in Luna or the deterministic Q8 TS-Guard. The lesson transfers; the magnitudes are model-specific.

Source: Shortcut Sensitivity in Agent-Safety Evaluations

Failure class two: the task was well-specified, the delivery was not

FDE-Gym, by Dongxu Yang and colleagues, measures the distance between those two worlds. It holds 95 end-to-end delivery tasks adapted by industry experts from real projects, spanning 20 customer scenarios across 18 sectors, in simulated environments with scripted stakeholders and modifiable customer systems. Scoring runs on the customer's own success metric, normalized from 0 (unchanged system) to 1 (reference solution), with penalties for messages, trial runs and waiting time. Across 13 configurations from seven providers, the best configuration averaged 0.56 and resolved only 14 of the 95 tasks, where resolution means a score of at least 0.8 with no incident. The authors identify discovering unstated constraints as the primary bottleneck, and the sharper result is the second one: on the 23 tasks where the best configuration obtained every key fact from stakeholders, it still reached 0.8 uplift on only 10. Knowing everything and delivering correctly are different competencies. The pre-shipment probe follows directly: take a handful of genuine tickets or scenarios from your own pipeline, strip the unstated constraints the original requester assumed everyone knew, and watch where the agent stalls. Those stall points are the incidents your well-specified suite will never produce. These figures are author-reported on that sample, not a universal ceiling, but they quantify a gap every builder meets anecdotally.

Source: FDE-Gym: The Last Mile of AI Delivery

Failure class three: extraction was right, interpretation drifted

SentX's public evidence page publishes a completed observation record from 3 October 2026 in which the full extracted text of a 25-page research preprint was supplied in one request and the answer asked for was an evidence brief. The method was disciplined: the reviewer visually checked the two load-bearing tables against the PDF before inference, nine frozen reference checks were withheld from the model, and the extraction's own warnings — column interleaving, unverified visual fidelity — were logged. The finished record includes the prompt, the answer, the source checks, the errors and qualifications, and the first-party review logged errors and qualifications against the delivered answer. Interpretation drift concentrates exactly where lossy compression hurts least visibly — in scope words, hedges and generalizations that no numeric check touches. The record's stated limits are part of its value: one paper, one answer, one first-party review, not an independent audit. It demonstrates the failure mode, not its frequency.

Source: Reading a complete research paper: one observed answer and source checks; Victoria: public model card

Ordering the investigation

When production breaks despite a green board, work the classes in cost order. First, run the perturbation screen on the harness that went green — it rules out the cheapest explanation, a shortcut the suite was scoring all along. Second, pull the failing production case beside the task spec that covered it: if the spec itself would have failed the case, you have a delivery gap, and no amount of model tuning closes what FDE-Gym shows even total information discovery leaves behind. Third, diff the agent's output claims against the source facts, reading for quantifiers, percentages and causal language rather than digits. Each step eliminates a class before you spend money on the next.

Source: FDE-Gym: The Last Mile of AI Delivery

The discipline already exists in miniature. SentX's model card states its own known limits plainly — answers can contain incorrect facts, citations or calculations, so open cited sources and check the supporting passage. A green report is a source too. Open it, find the supporting passage, and ask what representation it measured. The builder who applies that one habit to their own eval board is applying the same discipline the two October papers point toward — treat the number as a claim about a specific representation, then go read the representation.

Source: Victoria: public model card