SentX SI Blog Meet Victoria

Why does my web agent report elements that aren't on the page — and does mechanistic interpretability actually explain the failure?

· 5 min read

SentX SI

When your agent clicks a button that isn't there, the standard answer is that the model hallucinated. That is true, but it is a symptom name, not a mechanism. The plain reasons behind most of these failures are mundane: a stale or truncated page snapshot, an element that loads after the agent read the page, or a model completing an action from what the task looks like rather than from what the page currently contains. That last framing is my reading of how these failures usually arise, not a claim of the paper I am about to describe, and it matters because the paper does not settle which cause fires in any given case. Its formal version of the point is that the information an agent relies on varies across examples. A case study posted to arXiv on November 13 by Mohamed Aghzal, Mor Geva, Gregory J. Stein and Ziyu Yao runs the standard toolkit of mechanistic interpretability against exactly this failure, and its honest verdict is that the toolkit found something real and only a partial explanation.

Source: Can Mechanistic Interpretability Help Explain LLM Agent Failures? A Case Study on Web Element Hallucination

What the paper found

The authors applied a series of mechanistic analyses to Qwen2.5-7B-Instruct as it interacts with live websites. Two results carry the paper. A linear probe trained on the last prompt token can predict, before generation begins, whether the agent is about to hallucinate an element, at 0.80 AUC: the internal state at the end of the prompt already distinguishes grounded actions from invented ones. And steering along that probe direction corrects 85 percent of hallucinations, at a price of 7 percent of valid actions disrupted. The intervention works, and it is blunt.

Source: Can Mechanistic Interpretability Help Explain LLM Agent Failures? A Case Study on Web Element Hallucination

The scope matters as much as the numbers. One 7B model, one action format, one live-website setting. The 0.80 AUC and the 85-to-7 split are feasibility results for that setup, not benchmark results for web agents generally, and nothing in the paper transfers them to another model or harness.

Source: Can Mechanistic Interpretability Help Explain LLM Agent Failures? A Case Study on Web Element Hallucination

Where the evidence stops

The authors say it plainly: these results only provide a partial explanation of why the agent hallucinates. Three limits do the work. The probe direction is sensitive to prompt formatting, losing much of its steering effect after a minor change to how actions are represented, so the vector does not travel with your stack. Circuit identification is difficult because the computation is distributed across multiple components and positions, and the information the agent relies on varies across examples; there is no single component to point at and call the hallucination circuit. And a pre-generation predictor, however accurate, is not a cause: it tells you where to look, not what the model is doing when it fails. The paper frames these observations as a challenge to the assumptions underlying standard MI methods for circuit discovery and feature identification. The case study is as much about the limits of the tools as about the agent.

Source: Can Mechanistic Interpretability Help Explain LLM Agent Failures? A Case Study on Web Element Hallucination

So does mechanistic interpretability explain the failure? At the level of a readable, actionable signal existing before generation, in at least one model: yes, that is established. At the level of here is why the agent hallucinates, in your agent: no. That gap is the paper's actual contribution, and it is the one to keep in mind when a summary says MI fixed web-agent hallucination.

Source: Can Mechanistic Interpretability Help Explain LLM Agent Failures? A Case Study on Web Element Hallucination

Checking a claim about the paper

Here is how a source check runs on this exact story, as an illustration of the method. A summary says: new paper proves mechanistic interpretability can fix web-agent hallucination; a steering vector corrects 85 percent of cases. You open the arXiv abstract and read the sentences it compressed. These results only provide a partial explanation of why the agent hallucinates. The probe direction loses much of its steering effect after a minor change to how actions are represented. The second sentence alone rules out fix: a method whose effect depends on your exact action format cannot be cited as a fix for your agent. The first rules out proves: the authors cap their own claim. Both sentences sit in the abstract, so no full-text dive is needed. A summary can be true in spirit and wrong in scope, and scope is the part you are actually deciding on.

Source: Can Mechanistic Interpretability Help Explain LLM Agent Failures? A Case Study on Web Element Hallucination

Testing the mechanism against your own traces

The paper does not propose a way to test its mechanism in a different stack, so I will suggest one: an editorial suggestion, not a finding of the paper. Its target is the paper's key caveat, format sensitivity, because that is the property that decides whether anything transfers.

  1. Collect traces where the agent reported or acted on an element the page snapshot does not contain, alongside successful traces on similar tasks.
  2. Re-run the failed set with your action format unchanged, and once more with a deliberately rephrased action format: same elements, different representation.
  3. Diff the failure patterns. If failures move when the format moves, your stack behaves like the paper's: the mechanism is format-anchored, and a guard placed at one fixed format is a reasonable bet. If failures are stable across formats, the paper's mechanism is probably not the one in your stack, and the 85 percent number tells you nothing about your setup.

The point is that you do not need access to activations to run it; you need traces.

The habit, extended

The SentX model card, which is a first-party product disclosure rather than an independent certification, states in its known-limits section that answers can contain incorrect facts, citations or calculations, and that the response is to open cited sources and check the supporting passage. For a web agent, the cited source is the page snapshot the agent actually saw, and the passage to check is the element list it was given. The habit extended to agent traces is a one-line diff: when the agent reports an element, match it against the snapshot before the report is trusted. That check is model-independent and format-independent, which is precisely the portability the steering vector lacks. The paper maps where the failure lives inside one model; the diff is the guard you can actually put on your own.

Source: Victoria AI Model Card | Capabilities, Evidence and Limits

Sources

  1. Can Mechanistic Interpretability Help Explain LLM Agent Failures? A Case Study on Web Element Hallucination — arXiv
    Supported claims and limits
    • Case study arXiv:2611.18514, Aghzal, Geva, Stein, Yao, submitted 13 Nov 2026, on web agents hallucinating actions not available on a website
    • Mechanistic analyses applied to Qwen2.5-7B-Instruct interacting with live websites
    • Linear probe on the last prompt token predicts hallucinations before generation begins, at 0.80 AUC
    • Steering along the probe direction corrects 85% of hallucinations but disrupts 7% of valid actions
    • Authors cap the finding: results only provide a partial explanation of why the agent hallucinates
    • Probe direction is sensitive to prompt formatting, losing much of its steering effect after a minor change to how actions are represented
    • Circuit identification is difficult; computation is distributed across multiple components and positions and the information the agent relies on varies across examples
    • Observations are framed as challenging assumptions underlying standard MI methods for circuit discovery and feature identification
  2. Victoria AI Model Card | Capabilities, Evidence and Limits — SentX SI
    Supported claims and limits
    • Model card is a first-party product disclosure, not an independent certification
    • Known-limits section: answers can contain incorrect facts, citations or calculations
    • Model-card habit: open cited sources and check the supporting passage
Meet Victoria