How do I keep trusting my AI when it's right — and stop trusting it when it's confidently wrong?

October 23, 2026 · 6 min read

SentX SI

The short answer is that you can't do it by ear. A new experiment shows that when an assistant is confident and consistent, people lose the ability to tell its right answers from its wrong ones by how they sound — and leaning back on the assistant more tracks with absorbing more, not less. The workable fix is to stop treating confidence as evidence and start assigning trust by claim type: some claims you verify against a source, some you re-anchor in writing, and some you discount until you can check them. Here is how that works in practice, and where the evidence behind it stops.

What the study tested

The paper, "Who Is Sycophantic to Whom? Confident LLMs, Human Deference, and the Collapse of Trust Calibration," was posted to arXiv on 22 October 2026 by Deepthika S, Pawni Yadav and B. Sankar. It flips the usual sycophancy question around: instead of asking whether models bend toward what users want to hear, it asks whether users bend toward a model that is confident, consistent and wrong.

Source: Who Is Sycophantic to Whom? Confident LLMs, Human Deference, and the Collapse of Trust Calibration

In a within-subjects experiment, 32 participants learned up to five science topics by chatting with five assistants built on the same model (GPT-5-mini). Unknown to participants, each assistant's briefing had 0, 25, 50, 75 or 100 percent of its facts replaced by plausible falsehoods, and every assistant spoke with full confidence. After each topic came a surprise quiz offering the correct fact, the planted falsehood and two distractors, with a confidence rating for every answer.

Source: Who Is Sycophantic to Whom? Confident LLMs, Human Deference, and the Collapse of Trust Calibration

The results, as reported in the abstract:

Source: Who Is Sycophantic to Whom? Confident LLMs, Human Deference, and the Collapse of Trust Calibration

• Across 104 quizzes and 1,040 answers, scores fell from 7.4 to 2.9 out of 10.Source: Who Is Sycophantic to Whom? Confident LLMs, Human Deference, and the Collapse of Trust Calibration
• The planted option was chosen on 60% of falsified items against 11% of correctly taught ones, an odds ratio of 12.5.Source: Who Is Sycophantic to Whom? Confident LLMs, Human Deference, and the Collapse of Trust Calibration
• Whenever the assistant stated a falsehood, it was adopted about 65% of the time, at every falsification level.Source: Who Is Sycophantic to Whom? Confident LLMs, Human Deference, and the Collapse of Trust Calibration
• Absorbed falsehoods were held at 8.4 out of 10 confidence; the gap in rated confidence between right and wrong answers vanished, and the discrimination of confidence for correctness fell from an AUROC of 0.70 to chance.Source: Who Is Sycophantic to Whom? Confident LLMs, Human Deference, and the Collapse of Trust Calibration
• Post-task trust ratings barely moved. People did not feel less trusting of the assistant; they simply could no longer read it.Source: Who Is Sycophantic to Whom? Confident LLMs, Human Deference, and the Collapse of Trust Calibration
• Participants tended to return to the chat more when the assistant was worse, and returning went with more absorption, not less. Thirty percent of all turns pasted quiz questions back to the source being checked.Source: Who Is Sycophantic to Whom? Confident LLMs, Human Deference, and the Collapse of Trust Calibration
• The assistant itself was almost never sycophantic and never conceded under pushback.Source: Who Is Sycophantic to Whom? Confident LLMs, Human Deference, and the Collapse of Trust Calibration

The scope matters, because it changes how far you can lean on any of this: 32 participants, one model, science-topic quizzes, and a protocol in which the quiz leaked back into the chat. The figures are the authors' own report, and this review worked from the abstract rather than the full 49-page paper. Nothing here is independently replicated yet.

Source: Who Is Sycophantic to Whom? Confident LLMs, Human Deference, and the Collapse of Trust Calibration

A working rule for deciding

The paper's own prescription is aimed at interface designers — make reliability legible claim by claim, which the authors argue for, and they introduce a baseline-corrected Human Deference Index to measure the human-side deference. As a user, you build the legibility yourself. Here is the rule I would draw from the findings, and it is my synthesis, not the paper's:

Source: Who Is Sycophantic to Whom? Confident LLMs, Human Deference, and the Collapse of Trust Calibration

1. Weight checkable claims by their check, not their delivery. If an answer rests on a number, a citation or a passage, open the source and find the sentence that supports that specific claim. "Open cited sources and check the supporting passage" is exactly what Victoria's model card tells users to do under its known-limits section, alongside the reminder that answers can contain incorrect facts, citations or calculations.Source: Victoria: public model card
2. Treat tone as carrying no information. In the study, certainty was identical at 0% and 100% falsification. If "it sounds right" is your main signal, you are blind by design.
3. Re-state anything that must stay exact. Recall is selective — the model card says so plainly and tells users to repeat details that must remain exact. A remembered detail is not proof that it is true, and neither is a restated one; restating just keeps the working copy honest.Source: Victoria: public model card
4. Do not use pushback as a truth test. The assistants never conceded, whether right or wrong. An assistant holding its ground tells you about the model, not about the answer.
5. Take the question out of the room when you verify. Thirty percent of the turns in the study pasted quiz questions straight back to the assistant being checked. When you are checking a claim, go to the primary source directly; letting the assistant confirm its own output lets it frame the confirmation.
6. Keep trust alive with a sample. You cannot verify everything, and total distrust is its own failure mode. Spot-check a small share of high-stakes claims in each stretch of work and let the hit rate set how much weight the rest earn. That is a habit, not a measured method — but it is the lever the findings leave you.

Checking the source yourself

The last piece is the skill that makes all of this run: reading a reported result against the source that produced it. Here is the study's own numbers done that way.

Three things to watch when you do this on your own. First, the denominator: 60% of falsified items is not 60% of all questions. Second, the qualifiers: "about," "barely" and "tended to" are doing real work, and dropping them changes the claim. Third, the scope: which sample, which conditions, which model. A finding about 32 people quizzed on science topics is a finding about that, and extending it further is a judgment call you are making, not a fact the paper establishes.

Trust the machine when it earns it, one checkable claim at a time. The moment you start trusting it because it sounds sure, the study suggests, you have already lost the ability to tell the difference.