How do I predict whether my AI agent's safeguards will hold in production — without running every attack live?

October 24, 2026 · 6 min read

SentX SI

You can cut the live-attack budget dramatically by letting a language model play the role of your production environment. Instead of executing every adversarial scenario against real tools, you have a second model simulate the tool responses — command output, file contents, network replies — while the agent under test, its monitors, and your attack scripts all run unchanged against that simulation. You then spend a small audit budget on genuinely executed runs, compare the simulation's predictions against those ground-truth outcomes, and keep the setup only if it beats your current fallback. In the paper this method comes from, that fallback is a separate purpose-built sandbox, and on the environments where it was tested, the simulation won by a wide margin.

Source: [2610.31593] Simulated Control Evals

The method is called simulated control evaluation, abbreviated SCE, and it appears in "Simulated Control Evals" (arXiv 2610.31593), by Jeffrey Olmo, David Zhang, and Tyler Tracy, submitted in October 2026. It targets a specific failure mode of red-teaming: control evaluations measure whether safeguards can prevent a model from causing undesired outcomes even when it is intentionally trying to do so, but they are typically run in purpose-built control settings whose results get extrapolated to deployment. The paper's starting point is that safeguard effectiveness in deployment can depend on features of the environment and the tasks that those control settings do not represent. A smaller simulation that mirrors your actual stack is therefore worth more than a larger sandbox that does not.

Source: [2610.31593] Simulated Control Evals

Why the usual sandbox misleads

Most teams test safeguards once, in a clean controlled environment, and assume the result travels. The gap shows up in the parts of production the sandbox never modeled: which tools exist, what state they return, how tasks are phrased, what files are already there. If your monitor catches exfiltration attempts in the sandbox but production tasks route the same intent through a different tool, the sandbox score says nothing about the incident that will actually happen. SCE reframes the question from "did our agent survive the test" to "how accurately does this test predict our deployment," and it measures that accuracy the same way you would measure any forecasting system: against ground truth.

The method, in six steps

1. Describe the deployment surface. Write down the tools the agent can call, the kinds of files and state it sees, and the task distribution it actually handles. This description is what the simulator has to reproduce, so vagueness here becomes prediction error later.Source: [2610.31593] Simulated Control Evals
2. Put a language model in the environment's seat. When the agent calls a tool, the simulator returns a plausible response instead of executing anything. The agent, the monitors scoring its behavior, and the attack strategies aimed at it all run exactly as they would live.Source: [2610.31593] Simulated Control Evals
3. Set an audit budget. Decide what fraction of evaluation runs you will execute for real to obtain ground truth. The paper's headline numbers come from a 5 percent audit budget — a small slice of runs executed, everything else simulated.Source: [2610.31593] Simulated Control Evals
4. Calibrate, don't trust. For each monitor, compare its simulated verdict against the executed outcome on the audited slice and record the prediction error per monitor. Fix or discount the monitors that systematically miss; the point is to know each monitor's error before you lean on it.Source: [2610.31593] Simulated Control Evals
5. Run the baseline. Execute the same protocol in your existing separate sandbox under identical conditions. The comparison that matters is simulation versus separate sandbox, not simulation versus nothing.Source: [2610.31593] Simulated Control Evals
6. Close the loop. Attack strategies or monitor improvements discovered in simulation should be validated on held-out executed attacks before you rely on them. The paper reports that simulation-discovered strategies improved monitor performance on held-out attacks in its test environment, outperforming the separate sandbox and approximately matching the improvement from strategies found by executing.Source: [2610.31593] Simulated Control Evals

What the measurements actually show

The authors evaluated SCEs on LinuxArena, an environment where execution provides ground-truth outcomes, comparing prediction errors against BashArena, a separate control setting, and against variation in control metrics within LinuxArena itself. For independently sampled evaluation and deployment task sets, SCEs predicted safety with a root mean squared error of 3.6 percentage points across five monitors at the 5 percent audit budget, compared with 10.1 for BashArena. The simulated runs also reproduced the shape of what really happens: distributions of attack strategies, attacker reasoning, and monitor reasoning came out with over tenfold lower KL divergence from execution than BashArena's. And the simulation captured most of what differs between environments, explaining 78 to 85 percent of the between-environment variance in attack success, monitor scores, and safety. On cost, the edge over BashArena held even when the evaluation used fewer than 5 percent as many task-pair draws as the deployment task set contained.

Source: [2610.31593] Simulated Control Evals

An illustrative walk-through

To make the shape concrete, here is a hypothetical — every number below is invented, not from the paper or any product. Suppose you run a coding agent with shell access and have assembled 200 attack scenarios. Your old process executed 40 of them live in a generic sandbox each month. Under an SCE workflow you might simulate all 200, execute 10 (5 percent) as your audit, and score each monitor's simulated verdicts against those 10. If monitor A agrees with execution on 9 of 10 and monitor B on 7, you keep A's simulated verdicts and either fix or discount B. Then you run the same 10 audits through the generic sandbox and find it disagrees on 6. The simulation has earned its keep on this slice: broader coverage, closer to reality than the old baseline, at a fraction of the execution cost. You repeat the audit every cycle, because the moment your deployment changes — new tools, new task mix — the calibration expires and you start measuring error again.

Where the method stops

The authors are careful with their own conclusion: these results suggest that deployment-specific SCEs can be more predictive than control evaluations executed in separate control settings. That "can" is doing work. Accuracy is established on two environments — LinuxArena as ground truth and BashArena as baseline — both in the shell-and-file-system task family. Nothing in the paper measures how far the gap extends to a web application, a retrieval pipeline, or a multi-agent system, and the whole method leans on having an executable slice somewhere to calibrate against; with zero ground truth, you are trusting the simulator unmeasured. The 3.6 figure is also a property of that specific setup — five monitors, 5 percent audit, independently sampled task sets — not a constant you can carry into your own stack.

Source: [2610.31593] Simulated Control Evals

Two notes on what you feed the pipeline

If you build this around a hosted assistant, the simulator is itself a generative model, and the discipline the SentX SI model card asks of its own users transfers directly: answers can contain incorrect facts, citations, or calculations, so open the source and check the supporting passage — in your case, execute the audit slice and check each prediction against it. The privacy boundary is equally real. Attack scenarios, internal tool schemas, and production traces are exactly the material a simulation pipeline wants to ingest, and if any part of that pipeline is a hosted service, read its data practices before feeding it. SentX SI's published privacy policy, for instance, states plainly that submissions are not confidential and advises against submitting trade secrets, privileged communications, protected health information, financial account credentials, or any content you wish to keep private — a sensible default standard for deciding which parts of your eval stack stay in your own infrastructure.

Source: Victoria AI Model Card | Capabilities, Evidence and Limits | SentX SI; Privacy Policy | SentX SI (sentx.ai)