Two models score almost the same on benchmarks — how do I find the capability gaps the single number hides?
3 min read
When two models land within a point of each other on a headline score, the total has stopped telling you much. It averages every task into one number, and the averaging is exactly where the detail goes: a model can lead overall while trailing on the category your work actually lives in, and a near-tie at the top usually means the two systems are strong in different places. The practical answer is to stop reading the aggregate and start reading the rows beneath it — and the most informative thing to look for is not how large the gap is, but whether the ranking flips from one task family to another.
That move — decomposing one score into the behaviors behind it — is now a formal method. A paper called TIDE (Tree-Induced Fine-Grained Diagnostic Evaluation from Model Behaviors), posted to arXiv on October 21, 2026 under identifier 2610.29985, recasts diagnostic evaluation as behavior-driven decision-tree induction. Rather than accept the aggregate, TIDE recursively builds a hierarchical capability tree from benchmark samples and model scores: at each node, an LLM judge proposes candidate splits and performs sample-to-group classification, and a behavior-based information-gain measure selects among the candidates. The output is a tree whose branches name the specific capability contrasts the single number absorbed. The authors applied it to five models across four benchmarks and report fine-grained contrasts that are invisible at the aggregate level.
Source: TIDE: Tree-Induced Fine-Grained Diagnostic Evaluation from Model Behaviors
The clearest example is the contrast the abstract itself highlights: Claude Opus 4.6 achieves 100% on microeconomic market reasoning, where GPT-5.4 scores 83.3%, yet on contract law reasoning the order reverses, with Opus 4.6 at 77.8% against GPT-5.4's 88.9%. Depending on which row you read, one model is the clear leader and the other the clear laggard. Fold both sets of scores into a weighted total and the difference can shrink to noise, with the winner determined by how the weights happen to fall rather than by which system suits your task.
Source: TIDE: Tree-Induced Fine-Grained Diagnostic Evaluation from Model Behaviors
You don't need the tree machinery to capture most of the value. The paper's lesson translates into a reading method you can run on any published per-task table, offered here as an editorial suggestion rather than a procedure the paper tests: pick two comparable models evaluated under the same conditions and on the same date; group the task rows into families — reasoning domains, tool use, long-context work, whatever the table supports; then compare the rankings inside each family instead of the totals. Flag every family where the order flips relative to the overall ranking. Gap size matters secondarily; a reversal tells you the better model is a function of the task, which is the sentence you want in front of whoever makes the call.
The limits matter as much as the method. It can only surface what the table already contains — if the benchmark never tested the task family your work lives in, no reading trick conjures it, and the honest response is a small probe set of your own. Nothing here extends past the models and versions the table describes; a new release resets the question. And on the paper side, TIDE is a preprint: the results are author-reported and have not yet been independently reproduced, and the tree is shaped by the LLM judge doing the splitting, so it functions as a diagnostic lens on the data, not ground truth.
Source: TIDE: Tree-Induced Fine-Grained Diagnostic Evaluation from Model Behaviors
The same hygiene applies to every vendor's numbers, ours included. SentX SI's public model card states plainly that it is a first-party product disclosure and asserts no independently verified benchmark score; its known-limits section tells readers to open cited sources and check the supporting passage themselves. We hold our own figures to the standard this article recommends. A single aggregate score deserves the same treatment from anyone: read the rows, watch for the flip, and decide on the task, not the total.
Source: Victoria AI Model Card | Capabilities, Evidence and Limits | SentX SI
Sources
- TIDE: Tree-Induced Fine-Grained Diagnostic Evaluation from Model Behaviors — arXiv
Supported claims and limits
- TIDE recasts diagnostic evaluation as behavior-driven decision-tree induction for fine-grained capability diagnosis
- TIDE recursively builds a hierarchical capability tree from benchmark samples and model scores; at each node an LLM judge proposes candidate splits and performs sample-to-group classification, and behavior-based information gain selects among them
- Demonstration scope is five models across four benchmarks, author-reported; fine-grained contrasts found include Claude Opus 4.6 at 100% on microeconomic market reasoning where GPT-5.4 scores 83.3%, and Opus 4.6 trailing GPT-5.4 on contract law reasoning at 77.8% versus 88.9%
- Paper is arXiv v1 submitted 21 Oct 2026; inspected page states no venue or acceptance
- Victoria AI Model Card | Capabilities, Evidence and Limits | SentX SI — SentX SI
Supported claims and limits
- Model card is a first-party product disclosure, not an independent certification
- Card does not assert an independently verified underlying model version, model size, training-data inventory or benchmark score
- Known-limits section directs readers to open cited sources and check the supporting passage themselves