Why do my agents' group decisions stay wrong when one member holds the rare fact that matters?

October 29, 2026 · 6 min read

SentX SI

Your group's decisions stay wrong because the rare fact is attacked twice. The first attack happens when the discussion is compressed into a short shared memory: under either of the two natural rules for deciding what to keep, the surviving evidence ends up imbalanced — relevance-based selection readmits on-topic decoys, and rarest-first selection is no better than random on average — and accuracy follows that balance rather than the rule that produced it. The second attack happens after compression, inside the decision model itself, which gives extra weight to facts attributed to many agents. A new arXiv paper, "Rare, Relevant, and Still Wrong: How Memory Fails LLM Group Decisions" by Anouar Imel, Imad Charradi, Abderrahman Sghir and Khalil El Khammar, submitted 27 October 2026, separates these failures and measures them, and its central finding is uncomfortable for anyone tuning a multi-agent pipeline: accuracy follows the balance of evidence the memory keeps, not the policy that kept it. Swapping one selection heuristic for another is unlikely to rescue your results, because the heuristic was never the variable doing the work.

Source: Rare, Relevant, and Still Wrong: How Memory Fails LLM Group Decisions

This is also not the familiar case of a single assistant repeating its own stored mistake. It is a group-level failure in the machinery between agents: what survives compression, and how the survivor list is annotated. The study tests that machinery on hidden-profile tasks, the classic setup in which each group member privately holds part of the information and the correct answer only emerges when the private pieces are pooled. On such tasks both obvious selection heuristics look attractive, because how widely a fact is held is confounded with whether it helps: rarity tends to mark the private, potentially decisive information, and relevance is easy to score against the decision criteria.

Source: Rare, Relevant, and Still Wrong: How Memory Fails LLM Group Decisions

To untangle the confound, the authors built CrossProfile, a diagnostic benchmark of 60 templates with automatically verified answer keys, producing 480 tasks in which the ownership of supporting, decoy, and irrelevant facts is crossed around while the text of the facts is held fixed. Results span Qwen2.5-7B, Llama-3.1-8B, and two DeepSeek models. Both heuristics fail in predictable places.

Source: Rare, Relevant, and Still Wrong: How Memory Fails LLM Group Decisions

Selecting by relevance fails because decoys are on topic. A relevance filter cannot separate a genuinely supporting fact from one that is simply on topic with the decision criteria, so it readmits the decoys; in the classic hidden-profile configuration, even a perfect relevance filter loses 23 to 31 points to rarest-first selection. Selecting by rarity fails the other way. Rarest-first is no better than random selection on average, and it wins only in the configurations where the rare facts happen to be the useful ones — a condition you generally cannot verify before the pipeline runs.

Source: Rare, Relevant, and Still Wrong: How Memory Fails LLM Group Decisions

The third failure sits downstream of selection. Even when the right facts survive compression, the decision model treats attribution as evidence: facts credited to several agents carry more weight than facts credited to one. The paper calls this a form of social proof. Removing the attribution tags entirely, or reassigning them at random with line order fixed, raised accuracy in the classic configuration by 13 to 24 points — and that is a damaging finding, because teams usually add attribution tags deliberately, for traceability. The metadata you introduced to make the pipeline auditable is part of what steers the decision.

Source: Rare, Relevant, and Still Wrong: How Memory Fails LLM Group Decisions

There is also an upstream problem the memory never reaches. In natural discussions, the complete transcript yielded lower accuracy than no notes at all: the evidence is lost before any memory acts, in what gets said and recorded in the first place.

Source: Rare, Relevant, and Still Wrong: How Memory Fails LLM Group Decisions

Two limits matter before you act on any of this. The findings come from controlled benchmark tasks with verified answer keys, on the four models tested; they do not establish how your specific stack, or other models, will behave. And the point ranges describe the study's classic hidden-profile configuration, not universal constants. What the paper establishes within its tested scope is the diagnosis: the selection policy is the wrong dial to turn, and the evidence balance plus the attribution metadata are the dials that move accuracy.

Source: Rare, Relevant, and Still Wrong: How Memory Fails LLM Group Decisions

Here is a check you can run on your own pipeline, adapted from the paper's design. It is a proposed method, not a result the study measured on production systems. First, reconstruct the memory: take a recent decision your group got wrong and pull the exact compressed context the decision model received, lines in the order they arrived. Second, audit the balance: enumerate the surviving claims and mark which side of the decision each one supports. If the side that should have won is thinner in the memory than it was in what your agents actually held, your compressor is the culprit, and no change to the decision prompt will fix it. Third, run the attribution test: feed the same lines back through the decision step with the agent tags removed, or shuffled across lines with the order fixed, and compare outcomes across a batch of cases rather than one. A double-digit swing would point at social proof as a live factor in your results, and the fix the study points to is nearly free: delete the tags before the final call. Fourth, if your pipeline passes raw transcripts instead of curated notes, compare against a no-notes baseline; the study's natural-discussion result says the fuller record can be the worse input. If you paste discussion records into an assistant to help with the audit, note that submissions to SentX are not confidential and may enter training or shared memory, so share only material you are allowed to submit.

Source: Privacy Policy | SentX SI

The same selectivity shows up at single-assistant scale, and it is worth naming where it comes from. SentX SI's public model card for Victoria states plainly that recall is selective and exact retention is not guaranteed, and its advice is to repeat details that must remain exact. Read methodologically, that is a description of the mechanism CrossProfile measures at group scale: the retention process fades what it judges low-value, and in a group pipeline that judgment lands on a fact only one agent holds, so nobody notices until the decision comes out wrong. The model card's discipline therefore scales up almost unchanged — state the load-bearing facts explicitly and repeatedly, and verify what the shared memory actually kept instead of assuming it did. The card is a first-party product disclosure, not an independent certification, but it names the same failure mode the benchmark quantifies.

Source: Victoria AI Model Card

The practical summary fits in one line: stop optimizing which items your compressor keeps, and start auditing what the evidence balance looks like after it runs — then strip or shuffle the attribution tags before the decision model sees them. The rare fact that mattered was probably still in the room; your pipeline just made sure nobody counted it.