How do I watch a running agent reason in real time — and intervene before it finishes a mistake?
7 min read
Most supervision of an agent happens at the edges of a run. Before it starts, a gate decides whether the task gets escalated to a stronger model; after it ends, you replay the trajectory to find which input caused the miss. While the run is alive, the fast signals are usually silent, even though a trace of tens of thousands of tokens is unfolding right in front of you. The practical answer, drawn from a University of Central Florida paper submitted to arXiv on 19 October 2026, is to put a cheap checker on the visible trace and let it accumulate evidence in two stages: a reversible nudge first, an irreversible stop only if the bad behavior persists after the nudge. The paper, titled SOMA: Continuous Monitoring and Intervention for System Two Reasoning with System One Models, has no venue acceptance yet, and every number below is the authors' own measurement on their test setup. The design, however, is a pattern you can build around your own agent.
Source: SOMA: Continuous Monitoring and Intervention for System Two Reasoning with System One Models
Why the gap exists
Fast "System One" decision models — the paper's working example is Jev — take a piece of text and a natural-language question about it and return a calibrated probability in one quick call. Until now they have been used as gates: deciding what enters the reasoning model, then falling silent once it starts. Existing remedies for specific reasoning failures were each tied to one scenario, one signal and one action, and many required training or white-box access to the model. SOMA's move is to make the monitor continuous and the objective swappable: the same fast model and the same controller serve different failure modes, a new objective is just a new plain-text specification, and nothing requires training or access to the slow model's internals.
Step 1: Write the failure as an observable statement
The heart of the method is the monitor statement, and it follows a strict pattern: X has appeared in the reasoning, but the agent is still doing Y. The authors chose the form for two reasons. Both halves can be read off the visible text, unlike whether an answer is correct, which the monitor cannot judge. And the statement stops being true the moment the agent changes behavior, so later scans tell you whether your intervention worked. The two tested statements show the register. For overthinking: "The reasoning has reached a final answer and keeps re-verifying it without naming any specific remaining concern." For unanswerable questions: "The reasoning has noticed that the problem is missing necessary information or has contradictory conditions, but is still trying to produce a definite answer, for example by assuming values the problem does not give." For your own deployment, write the failure you most want caught in that shape — an agent that has already found the needed figure in a retrieved document but keeps re-querying it, for instance. That second example is mine, not the paper's; the validated part is the form, and the form is what does the work.
Step 2: Scan at regular pauses while the run proceeds
In the paper, the slow model pauses every 256 thinking tokens — extended to the next line break, at most 128 further tokens — and at each pause the checker receives the task, the trace so far and the statement, and returns one probability. Each scan is a single short call, which is why it can run continuously without dominating the budget. The interface requirement is deliberately loose: any model that maps a text state and a natural-language statement to a probability can fill the role. The checker sees only what the slow model wrote itself; inserted reminders and system prompts are hidden from it, so its judgment always concerns the agent's own behavior. When you build this around your own pipeline, the pause points are whatever your harness exposes — step boundaries, message boundaries, a fixed token interval. The cadence is yours to choose; 256 tokens is a tested starting point, not a law.
Step 3: Escalate in two stages
This is the part worth copying most carefully. A single confident scan reaches at most the first rung — it cannot trigger the stop by itself. The controller accumulates evidence one-sidedly: a confident scan weighs more than a doubtful one, an isolated spike is absorbed by later low readings, and only behavior the checker keeps observing drives the score upward. When the score first turns positive, the response is a reminder — a short message inserted into the agent's own reasoning. It does not command; it points at the possible problem and asks the agent to judge whether its current activity is still useful, leaving the decision to the agent. It costs a few tokens and is fully reversible: the agent can dismiss it and carry on. Only if evidence keeps building after the reminder, past a fixed threshold, does the controller execute the takeover — predefined text that ends the thinking phase and steers the agent toward a final answer or a refusal. The agent still writes the closing content, but it cannot undo the stop. The ladder exists so the intervention lands at the earliest point that still changes the outcome: a nudge if the agent will respond, a hard stop only if the misbehavior demonstrably persists after it.
Step 4: Calibrate the stop before deployment, and accept what it costs
A reminder gone wrong is harmless; a takeover gone wrong is a wrong answer or a false refusal that never recovers. That asymmetry is why the threshold is not guessed. The authors calibrate on a small set of representative tasks before deployment, comparing each candidate threshold against the same agent running under the same setup without any intervention, so that every difference is caused by the intervention itself. Harm means tasks the intervention turns from correct to wrong — answers spoiled by an early stop and correct answers turned into refusals. They then pick the most aggressive threshold, the one that stops earliest and saves the most, whose harm stays within a tolerance you set, and freeze it for use. The tolerance is an honest knob: it is how much spoiled work you will accept in exchange for catching real failures. The price shows up in their numbers. On unanswerable questions, the authors report abstention rising from 70.5% to 87.2% on one model — the highest of all methods they tested — while false abstentions on answerable questions rose from 2.4% to 8.0% on one model and from 2.1% to 4.8% on the other. You are buying more correct refusals with some correct answers refused. Whether that trade is acceptable is a property of your workload, not of the method.
Step 5: Know exactly what the evidence covers
The authors' headline results: on overthinking, up to 36% fewer reasoning tokens with average accuracy slightly improved (91.0% to 91.3% on one model, 84.4% to 84.8% on the other); on unanswerable questions, the abstention gains above plus token cuts of 32% and 48%. Their ablations show the two stages doing different jobs: reminders alone already shorten overthinking without hurting accuracy, while on unanswerable questions the abstention gain comes from the takeover stage. One case study is worth keeping. On a question with a missing term, the model named the gap immediately, then answered under an assumption anyway after 6,148 tokens; under SOMA the same doubt became an abstention after 1,392. The scope limits, stated by the authors themselves, matter as much as the results. Two scenarios, both mathematical or scientific reasoning; two 8-billion-parameter models; one monitor per scenario; a single fast checker, Jev, a hosted proprietary model queried through a paid API at every pause. No other checker was tested, so the results do not establish which other models can fill the role; wall-clock latency was not measured; and the reported token counts cover the slow model only. Two practical consequences follow. Do not assume transfer to larger models, other domains, or several monitors running jointly — the authors flag all of that as untested. And if your traces contain private data, a hosted checker means sending that reasoning to a third party; a locally deployed open-source checker avoids it.
Source: SOMA: Continuous Monitoring and Intervention for System Two Reasoning with System One Models; SOMA: Continuous Monitoring and Intervention for System Two Reasoning with System One Models (full text)
Where this sits, and who is already asking you to do this
Pre-deployment audits tell you what an agent knows and where it should stop. Post-run attribution tells you which retrieved input produced a finished mistake. Neither acts while the run is alive. This pattern does: it names the failure in advance, watches for it cheaply and continuously, and spends its irreversible authority only after the agent has been given — and missed — a chance to fix itself.
The same instinct is already in plain view on SentX's own model card, which lists among its known limits that answers can contain incorrect facts, citations or calculations, and directs readers to open cited sources and check the supporting passage. The card describes itself as a first-party product disclosure, not an independent certification, and that is exactly how this belongs: the published guidance treats checking as the reader's standing obligation, and the pattern above is a way to discharge it while a run is still happening, with the false-stop cost priced in advance rather than discovered afterward.
Source: Victoria AI Model Card | Capabilities, Evidence and Limits
Sources
- SOMA: Continuous Monitoring and Intervention for System Two Reasoning with System One Models — arXiv
Supported claims and limits
- Paper is arXiv:2610.26793v1, submitted 19 Oct 2026, by Qian, Shi, Hu, Yan, Wang and Tian; no venue acceptance appears on the page
- Abstract reports cutting reasoning tokens by up to 36% on overthinking while improving average accuracy, and raising abstention from 70.5% to 87.2% on unanswerable questions
- SOMA: Continuous Monitoring and Intervention for System Two Reasoning with System One Models (full text) — arXiv
Supported claims and limits
- S1 models such as Jev return a calibrated probability in one fast call and had been used only as gates before reasoning; existing remedies tied one signal to one action, many needing training or white-box access
- Monitor statements follow the pattern 'X has appeared in the reasoning, but S2 is still doing Y', chosen because both halves are readable from visible text and the statement stops being true once behavior changes
- Both tested monitor statements quoted in the article match Table 1 verbatim
- S2 pauses every 256 thinking tokens, extending to the next line break for at most 128 further tokens; each scan is one short call; any model mapping a text state and statement to a probability fits the interface; reminders and system prompt are hidden from S1
- Controller accumulates one-sided log-odds evidence; first positive score inserts a reminder that costs a few tokens and leaves S2 in charge; takeover is predefined text ending thinking, cannot be undone by S2, and fires only on persistent post-reminder evidence
- Thresholds are calibrated before deployment on representative tasks against the same setup without intervention; the rule selects the most aggressive threshold whose harm stays within a tolerance, then freezes it
- False abstentions on answerable questions rose from 2.4% to 8.0% and from 2.1% to 4.8% on average across the two S2 models
- Reminders alone already shorten overthinking without lowering accuracy; on unanswerable questions the abstention gain comes from the takeover stage
- Case study: Qwen3-8B names a missing term at once, answers under an assumption after 6,148 tokens without SOMA, abstains after 1,392 tokens under SOMA
- Scope limits: two scenarios (mathematical/scientific reasoning), two 8B models, one monitor per scenario, Jev as sole S1 via paid API at every pause, no other S1 tested, wall-clock latency not measured, token counts cover S2 only; transfer to larger models, other domains or joint monitors untested
- Relying on a hosted S1 sends reasoning to a third party, which matters for private data; a locally deployed open-source S1 avoids this
- Victoria AI Model Card | Capabilities, Evidence and Limits — SentX SI
Supported claims and limits
- Model card lists as a known limit that answers can contain incorrect facts, citations or calculations
- Card directs readers to open cited sources and check the supporting passage
- Card identifies itself as a first-party product disclosure, not an independent certification