How do I know my agent-safety evals measure real safety and not just rewards for shortcuts?

October 17, 2026 · 4 min read

SentX SI

You find out by deliberately breaking your own benchmark. Hold the safety property you care about fixed — say, a guard blocking sensitive tool calls — and change only the surface features of the scenarios that test it: the wording of instructions, the order of steps, the distractor details, the names of the available tools. If your green scores move systematically with those cosmetic changes, the suite was scoring the shortcut, not the capability. That logic now has a published audit behind it. In Shortcut Sensitivity in Agent-Safety Evaluations (arXiv 2610.22243), submitted in October 2026 and accepted at the NeurIPS 2026 workshops TAE and AIWILD, Hasan Demirkiran and Jens Ernstberger show that the data used to evaluate a guard can carry cues that make labels predictable without measuring the intended safety construct, and that the same cues can steer the guard's own decisions. Their framework separates three checks: whether labels are predictable from surface features, whether guard behavior is merely associated with observable markers, and whether paired decisions change when you intervene on those markers. Here is the pre-flight version of that audit, in four moves you can run on your own harness before you trust the dashboard.

Source: Shortcut Sensitivity in Agent-Safety Evaluations

Move one is a label-predictability screen. Before evaluating anything, ask whether your benchmark's labels are predictable from features that have nothing to do with safety. In the LinuxArena trajectories the authors audited, shorter average shell commands strongly predicted the trajectory labels — a purely mechanical statistic was carrying the answer. On your own suite, compute the obvious surface statistics: command length, step count, tool-name frequency, boilerplate phrasing. See how well they separate safe from unsafe rows. If a trivial rule approaches your guard's accuracy, the dataset is leaking, and every score computed on it overstates coverage.

Source: Shortcut Sensitivity in Agent-Safety Evaluations

Move two is domain group holdouts instead of random row folds. Random cross-validation shuffles rows, so training and test rows from the same scenario family sit side by side, and a model that memorizes domain-specific surface patterns still passes. Under domain holdout, where whole scenario families are withheld, exact-tool lookup F1 on the AgentDojo-Traj dataset fell from .901 with mixed-domain folds to .495. That is not a small wobble; it is the difference between a measurement and a mirror. Re-run your standard folds grouped by domain and watch what survives.

Source: Shortcut Sensitivity in Agent-Safety Evaluations

Move three is evaluation-unit metrics instead of row-weighted F1. When one domain contributes ten times the rows of another, row weighting measures the big domain. On AgentHarm-Traj, the released TS-Guard's F1 fell from .902 under row weighting to .719 with equal interaction weighting, while balanced accuracy rose — the headline number had been doing the heavy lifting. Report unit-weighted metrics and per-domain breakdowns alongside the aggregate, so a strong domain cannot hide a weak one.

Source: Shortcut Sensitivity in Agent-Safety Evaluations

Move four is placebo-controlled paired perturbations, the most informative of the four. Build minimal pairs: two scenarios identical except for one cue — a marker rewritten, a tool renamed to something opaque, a step reordered, a distractor added. Run your guard on both, repeatedly and interleaved, so ordering and drift cannot explain the difference. Then subtract a formatting placebo, an edit that changes nothing meaningful, to separate representation effects from generic perturbation noise. The paper's anchor result came from exactly this design: on a frozen 64-step cohort from AgentDojo-Traj, screened across seven configurations with repeated, interleaved Haiku calls, rewriting the marker lowered blocking by 8.33 percentage points, and swapping tool names for opaque aliases raised it by 11.46 points. All unsafe calls remained blocked in both cases; the movement was false positives against inherited labels. Both effects stayed nonzero after placebo subtraction, with exploratory 95 percent ranges excluding zero, and the same edits shifted measured accuracy against the inherited labels as well. The benchmark's packaging, not the underlying safety behavior, had moved the score.

Source: Shortcut Sensitivity in Agent-Safety Evaluations

Two caveats keep this honest. The Haiku-sized effects did not reproduce in Luna or in the deterministic Q8 TS-Guard, so if your guard is rule-based or deterministic, expect less shortcut sensitivity — which tempers the alarm rather than removing the need to check. And the audit covers completed LinuxArena trajectories and two TS-Bench-derived step datasets; the protocol transfers to your harness as a method, and your numbers will be your own. Domain group holdouts, evaluation-unit metrics, and placebo-controlled paired interventions are the authors' stated recommendations; the label-predictability screen is the cheap first filter that tells you how urgent the other three are.

Source: Shortcut Sensitivity in Agent-Safety Evaluations

The discipline behind all four moves is the same one any careful reader applies to a report: do not accept the aggregate, open the underlying record and verify what produced it. SentX's own model card puts that instruction in its known-limits section — open cited sources and check the supporting passage — and an eval dashboard is just another kind of citation. The green number is the claim; the trajectories behind it are the source. A suite whose scores hold under domain holdouts, unit weighting, and placebo-controlled perturbations is one you can defend to an auditor. A suite that moves under those tests is telling you, precisely and cheaply, what it has actually been measuring all along.

Source: Victoria: public model card