My agent gets the right answer sometimes and wrong others — how do I find which tool result caused the miss?

October 20, 2026 · 6 min read

SentX SI

When the same agent answers the same kind of question correctly in one run and wrongly in the next, the fault is rarely where you look first: the computation. A paper submitted to arXiv on October 19, 2026, "Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy," by Ke Zhang and colleagues, documents agents that computed the right value and still answered wrongly, because they answered from the wrong piece of evidence. The useful takeaway is a way to find out which of the three possible failures happened in your own runs.

Source: Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy

The setting is now common. Agent harnesses increasingly keep long tool outputs out of the context window, saving them to files that the agent searches or reads on demand. Once that happens, a wrong answer on its own cannot tell you whether the tool produced the wrong value, whether the right value never reached the agent, or whether the agent saw it and still botched the answer. The paper's method splits every miss into those three stages and asks each one separately.

Source: Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy

Build: did any of the agent's own tool runs actually print the keyed value? Exposure: given that it was produced, did it reach the agent, inline or through a read or search result? Closure: given that the agent saw the value, did it answer correctly from it? Alongside the stages, the authors record the file from which each seen value came. Each stage points to a different remedy: a build defect argues for better inputs to the tool, while an exposure or closure loss argues for changing what and how the agent reads.

Source: Same Simulation, Different Evidence (full text)

What makes the method workable at all is determinism. The study uses PHREEQC, a widely used USGS geochemistry simulator whose section-structured reports reach 846.8 thousand characters and whose determinism lets every printed value be traced. The agents answered 158 multiple-choice items whose keys the authors re-verified by execution. They rebuilt and re-ran all 11,007 logged simulator calls and searched every tool result each agent received, so each trajectory got a ground-truth label at every stage without a judge.

Source: Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy

The result is model-dependent, which is itself a warning. Reading the simulator's report through a section index rather than inline costs GPT-5.1 eleven point three and GPT-5.2 seven point nine accuracy points, while Claude Opus 4.6 and Claude Sonnet 4.6 show no detectable cost. GPT-5.4's net change is indistinguishable from zero, but that null hides roughly ten points of evidence-use loss masked by more frequent correct simulations. Across the board, re-running the logged calls shows the GPT models compute the keyed value about as often under retrieval as under inline delivery; what changes is the evidence they answer from.

Source: Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy

The failure mode has a name in the paper: evidence substitution. Under retrieval, the GPT agents skip the canonical report in most trajectories and instead read their own selected-output files, which print the same quantities with more digits. They then truncate those longer numbers to a neighboring option. In the trap cases, where a value rounds to the correct answer but truncates to a distractor, GPT-5.2 picked the distractor in 26 of 35 trajectories and GPT-5.1 in 19 of 27, against 1 of 35 for Sonnet.

Source: Same Simulation, Different Evidence (full text)

The paper's worked example is item 121, a pH after the first reaction step, with options 2.750, 2.751, 2.748 and 2.745 and the key at 2.751. The simulator's report prints pH = 2.751 at line 120, at the options' precision; the agent's own output file prints 2.75065. Under the section-index condition, the agent reads only its own file, notes that the value rounds to 2.750, picks the wrong option, and never opens the report, even though the index pointed at the section containing the keyed value. It does this even when told how to round. Told instead to answer from the report, it answers correctly.

Source: Same Simulation, Different Evidence (full text)

The authors then confirm the location by holding computation fixed. They freeze GPT-5.2's logged inline run at its first successful execution on 131 numeric items and continue the identical visible history twice: once with the report inline, once with the section index. Both arms share the files, the system prompt, the reading tools and the remaining turns; only the delivery changes. Over three paired repeats, the index arm costs 9.4 points. Same simulation, different evidence, same loss.

Source: Same Simulation, Different Evidence (full text)

Interventions close the gap. Naming the authoritative output, telling the agent to take the value from the relevant section of the report and not from its own selected-output files, raises accuracy by 6.3, 7.9 and 7.8 points for GPT-5.1, GPT-5.2 and GPT-5.4 respectively, cuts report skipping to around zero, and restores closure to about 99 percent. Inlining reports that fit within the budget works as well for the two older GPT models. Merely instructing the agent to round correctly, with no file named, does not: accuracy moves less than a point and half, and the agents still pick the truncated neighbor in most trap cases. The lesson is that directing the agent to the value already printed at the right precision works where asking it to do arithmetic on a value it grabbed from the wrong place does not.

Source: Same Simulation, Different Evidence (full text)

Applied to your own inconsistent agent, the method becomes a pre-mortem checklist. First, start logging which file each value the agent cites came from; if your harness does not capture that, nothing downstream is auditable. Second, suspect first any output the agent wrote itself, especially higher-precision scratch files that compete with a tool's canonical report. Third, check whether the canonical output was ever actually opened, not merely listed in an index. Fourth, scan your wrong answers for values that equal a truncation of a longer number you can find in the logs; that pattern is the fingerprint of evidence substitution. Fifth, test the location the way the paper does: re-run the same tasks with the authoritative output named explicitly or inlined, holding everything else fixed, and see whether accuracy jumps. If it does, your loss was in exposure or closure, not in the tool, and the fix is a delivery change, not a model upgrade. These steps are the paper's logic applied beyond its own benchmark, not a measured result in your setting.

Source: Same Simulation, Different Evidence (full text)

Scope matters. The study covers one deterministic tool, one benchmark of 158 items, six closed API models accessed at default or zero-temperature settings, one wording of each instruction, and a paired replay that covers GPT-5.2 only. The stage decomposition allocates an observed difference among the stages rather than estimating independent causal effects, and the authors do not determine which model property drives the split between the GPT models and the Claude models. Treat the specific magnitudes as what they are, measurements on this setup, and expect to re-validate per model before you trust the pattern in your harness.

Source: Same Simulation, Different Evidence (full text)

The same principle shows up in how we document our own product. Victoria's public model card states as a known limit that answers can contain incorrect facts, citations or calculations, and the practical direction attached to it is to open cited sources and check the supporting passage. That is the user-side version of the paper's best intervention: point at the authoritative artifact instead of trusting the number already in hand. We publish it as guidance for readers, not as a guarantee of accuracy, which is exactly the scoping the research supports.

Source: Victoria AI Model Card | Capabilities, Evidence and Limits | SentX SI