How do I check that an AI's proposal from my team's opinions is stable — and not hiding options it dropped on a rerun?

November 4, 2026 · 4 min read

SentX SI

The short answer: feed the same set of opinions to the mediator more than once, pull the assignments out of each draft, and diff them. If an assignment appears in some runs and vanishes in others, the single proposal you were about to publish is one of several the system silently chose between. A new preprint, "From Opinions to Commitments: Unstable Assignments and Silent Choices in AI-Mediated Proposals" (Mahajan, Argota Sánchez-Vaquerizo, Capozzi, Helbing; arXiv:2611.05135, submitted 3 November 2026), measures exactly this and finds the instability is real, survives the usual fixes, and stays invisible if you only ever see one draft.

Source: From Opinions to Commitments: Unstable Assignments and Silent Choices in AI-Mediated Proposals

What the study did

Large language models are increasingly used to turn citizens' opinions into collective proposals, and those proposals make assignments: who should act, what must be done, how much one should spend, and by when. The authors tested four LLM mediators — three open-weight and one commercial — on opinions drawn from DeliberationBank and Polis. For each task, a mediator drafted a proposal from thirty opinions five times, varying either the random seed or the order of the opinions. That design isolates the question that matters for a team: if nothing changed except the run, did the proposal change?

Source: From Opinions to Commitments: Unstable Assignments and Silent Choices in AI-Mediated Proposals

What it found

In 17% of five-run sets from the open-weight mediators and 20% from the commercial one, some runs contained assignments while others did not — roughly one task in five produced a materially different commitment depending on which run you happened to publish. Reordering the opinions changed the assignments almost as often as replacing them with opinions from different citizens, so the instability is not only about who is in the room; it is about how the same room is fed to the model. Three human annotators confirmed that the detected differences between runs were real, although an LLM judge used to flag them overstated how often they occurred — a reminder that the diff needs human eyes, not just a second model. Substantial variation remained under four safeguards the authors tried: temperature zero, a commercial mediator, draft selection, and draft merging. And the mediator's objective shaped what got covered: prompts seeking agreement covered minority positions much less often than prompts asking to describe differences.

Source: From Opinions to Commitments: Unstable Assignments and Silent Choices in AI-Mediated Proposals

This is a different failure mode from the two adjacent ones. It is not whether an agent's published promise will hold up once made, and it is not whether one member's rare fact survives the group's deliberation. It is the mediator's own output stability: the same input, run twice, can commit the team to different things.

A repeat-run stability check

The paper proposes audits that make alternative assignments visible and disclose the objective behind each proposal. As a working version of that idea for a team that does not want to build an audit pipeline, here is a check we would propose — ours, not a finding of the paper:

Source: From Opinions to Commitments: Unstable Assignments and Silent Choices in AI-Mediated Proposals

1. Fix the input. Keep the opinions identical in content and record their order.
2. Rerun. The paper used five runs; use at least that many, varying the seed or the order as the paper did.Source: From Opinions to Commitments: Unstable Assignments and Silent Choices in AI-Mediated Proposals
3. Normalize. Extract the assignments from every draft into one comparable form: who, what, how much, by when.
4. Diff. Mark every assignment that appears in some runs and not others, and every position covered in some drafts and excluded in others.
5. Set your threshold. The paper's proposal is visibility, not a number, so the bar is yours: at minimum, any assignment that flips between runs on a task you are about to commit to is a reason to stop.Source: From Opinions to Commitments: Unstable Assignments and Silent Choices in AI-Mediated Proposals
6. Publish accordingly. If drift crosses the bar, publish the alternatives alongside the pick, and state the objective the mediator was prompted with — agreement-seeking, or describing differences — because that choice quietly decides which minority views make it in.Source: From Opinions to Commitments: Unstable Assignments and Silent Choices in AI-Mediated Proposals

Where the check stops

The study is a preprint at thirty opinions per task and five runs per task, so the percentages describe feasibility at that scale, not a benchmark for your deployment; and they are shares of five-run sets, not of individual runs. Your own drift rate will differ with the mediator, the task and the prompt. The check tells you that a pick is unstable, not which of the alternatives is right — that remains a human decision, which is precisely the point of making the alternatives visible.

Source: From Opinions to Commitments: Unstable Assignments and Silent Choices in AI-Mediated Proposals

The habit that makes the check possible

The check only works if the mediator's intermediate output is something you can actually open. That is the same habit SentX's model card states plainly in its known-limits section: answers can contain incorrect facts, citations or calculations, so open the cited source and check the supporting passage. For an AI-mediated proposal, the cited source is the full option set the mediator saw before publishing one pick — the other drafts, the dropped assignments, the prompt objective. A single published proposal is only as checkable as the material behind it, and the model card is a first-party disclosure of that practice, not an independent certification of it. If you cannot see what the mediator considered, you cannot verify what it chose; the stability check is how you find out whether that matters.

Source: Victoria: public model card