My small model passes its pass/fail evals — am I measuring reasoning, or just a memorized procedure?
8 min read
The short answer, from the cleanest demonstration of the problem I have seen in a while: a comfortable pass rate usually measures the shape of your training distribution, not generalizable judgment — and a paper submitted to arXiv on November 16 proves it by using a verifier with no error in it at all. "Procedure without Judgment: What Small Language Models Learn under an Exact Verifier" (arXiv:2611.21529) removes every confound that normally lets a good score hide a narrower claim. When the scorer cannot be wrong about whether an answer is right, whatever the score fails to tell you is exactly what pass/fail fails to tell you in general.
How the question becomes answerable
The task is explaining the decisions of an authorization policy. GlassGate, the paper's test gate, is a hand-written policy over ten categorical features — the tool being called, scope level, the target domain, content flags for personal, credential and financial data, the caller's authorization level and role, the declared intent, and prior violations — with 124,416 possible contexts and nine rules, eight blocks over an allow default. Because the space is enumerable, an oracle computes the exact minimal sufficient reason for every decision: the smallest set of features whose values force the outcome, with nothing left over. An explanation is scored against ground truth, not a judge model, not a rubric, not a heuristic.
Source: Procedure without Judgment: full text, arXiv:2611.21529v1
The model never sees the policy. It sees one request, one decision, and thirty turns in which it can probe the policy on altered requests or ask whether a candidate set of features is sufficient. Finding the minimal set is an experiment, and the paper asks who can run it: four frontier models, a small student fine-tuned on algorithm traces, a small student fine-tuned on frontier-model traces, and reinforcement learning trained on the exact reward.
Source: Procedure without Judgment: What Small Language Models Learn under an Exact Verifier (arXiv:2611.21529, abstract page); Procedure without Judgment: full text, arXiv:2611.21529v1
The 92 to 48 split
Qwen3-4B, fine-tuned on 2,000 traces of a deletion algorithm — a procedure that removes one feature at a time and keeps only what is needed — reaches 92% exact minimality on GlassGate. On its own gate, that is a success story. Then the same model meets three public policies it never saw, an OPA load-balancer rule and two Cedar policies with three to ten features each. Sufficiency holds at 94%. Exact minimality falls to 48%.
Source: Procedure without Judgment: What Small Language Models Learn under an Exact Verifier (arXiv:2611.21529, abstract page); Procedure without Judgment: full text, arXiv:2611.21529v1
That gap is the answer to your question. The model learned to verify that its cited set forces the decision, and it did not learn to search for the smallest set that does. The verification step travels to unfamiliar feature spaces; the search step stays home. On the same 47 off-distribution episodes, Haiku 4.5 with a one-sentence instruction to shrink its answers is exact on all of them. The 92% was never evidence of reasoning. It was evidence of a schedule fitted to ten features, run on ten features.
Source: Procedure without Judgment: What Small Language Models Learn under an Exact Verifier (arXiv:2611.21529, abstract page); Procedure without Judgment: full text, arXiv:2611.21529v1
A mirror experiment rules out the easy escape. A student taught the same deletion algorithm on the public policies only, never seeing GlassGate, reaches 92% on those policies and 28% on GlassGate. The recipe installs a schedule for whatever gate it was taught on, and whichever direction you run the experiment, the schedule stays home.
Source: Procedure without Judgment: full text, arXiv:2611.21529v1
Teacher kind and volume, separated
At matched data size — 148 traces each — the algorithm-taught student leads the frontier-taught student in all three training seeds, by 16, 47 and 61 points, and the worst algorithm seed beats the best frontier seed. The two arms never overlap. Scaling from 148 to 2,000 algorithm traces adds a further 23 points, so kind and volume both contribute, and the design separates the two: at matched size the kind gap holds in every seed, and with the teacher held fixed, volume adds its own gain on top.
Source: Procedure without Judgment: What Small Language Models Learn under an Exact Verifier (arXiv:2611.21529, abstract page); Procedure without Judgment: full text, arXiv:2611.21529v1
There is a floor worth knowing about. A single-flip heuristic — change one feature at a time, cite whatever flips the decision — scores 52% on the same gate. The frontier-taught student, at 53%, does not clear it, before or after reinforcement learning. If your small model was trained on traces of a bigger model solving your task and it passes, it may have reproduced the trace form without the semantics inside it, and your pass rate may be sitting just above a heuristic you never measured. The paper points to Valmeekam and colleagues (2025) for the mechanism: students reproduce intermediate token structure without the reasoning that generated it.
Source: Procedure without Judgment: full text, arXiv:2611.21529v1
The RL result: the score stayed, the strategy inverted
Warm-started from the frontier-taught student and trained against the exact reward, held-out accuracy moves three points, from 53% to 56%, with a p-value of 0.80. Not significant. But the strategy flips completely: probes drop to zero, and sufficiency checks go to 30 — the entire budget — on every episode. The model stops exploring and becomes an exhaustive verifier with the deletion algorithm's query shape and none of its judgment about which check to run next. A second warm start under identical hyperparameters diverged instead, falling to 6% exact minimality. One run went up three points, the other collapsed. Both share the inversion, and neither improved the answer.
Source: Procedure without Judgment: What Small Language Models Learn under an Exact Verifier (arXiv:2611.21529, abstract page); Procedure without Judgment: full text, arXiv:2611.21529v1
The authors' framing is blunt: the oracle made the effect of training legible, but it did not make training better. If your eval reports only a final pass/fail number, you see 56% and call it flat. You do not see that the model stopped probing, that it is spending its entire budget on one kind of check, or that a different seed would have given you 6%. The score established that the model is not getting better, and silently dropped everything else.
Source: Procedure without Judgment: What Small Language Models Learn under an Exact Verifier (arXiv:2611.21529, abstract page); Procedure without Judgment: full text, arXiv:2611.21529v1
Check the worked episode yourself
Appendix A.5 of the paper gives one GlassGate episode you can verify by hand. The request: an email.send call, read scope, target public.web, payload containing credentials and PII but no financial data, no authorization, analyst role, destructive intent, two prior violations. Decision: blocked. The oracle's minimal reason is three features — tool, target domain, contains credentials — because the policy blocks when an outbound channel carries credentials to a public domain.
Source: Procedure without Judgment: full text, arXiv:2611.21529v1
The deletion algorithm runs ten checks, one per feature, dropping each that is not needed, and lands on the three-feature set. The algorithm-taught student reproduces that pattern exactly. Sonnet 5, given the same tools, runs eight probes and two checks: it finds that flipping credentials unblocks the call, finds that flipping the target domain also unblocks it, and then commits to an empty set. It had the raw material for the answer and never assembled the conjunction.
Source: Procedure without Judgment: full text, arXiv:2611.21529v1
That episode is a template for your own gate. Take one passing output from your small model. If your task allows exact computation, you already know the ground-truth minimal set. Read the trace. Did the model arrive through a search that could generalize, or through a fixed sequence anchored to your specific feature count and structure? If the latter, the pass rate is measuring schedule fidelity, not reasoning.
Source: Procedure without Judgment: full text, arXiv:2611.21529v1
A perturbation check you can run on your own eval
The next idea is mine, not the paper's — a proposed method that follows the paper's off-distribution design. Take two or three tasks structurally similar to your eval gate but unseen in training: same task type, different feature names, a different feature count, or a reordered rule set. Score each output on two dimensions, separately.
Sufficiency: does the cited set force the correct decision? Minimality: is it the smallest set that does?
Read the gap. If sufficiency stays high and minimality drops, that is the paper's 94-to-48 pattern: your model has memorized a verification schedule, not a search procedure. If both stay high, the procedure is more likely to travel. If both drop, your in-distribution pass rate was format compliance or lucky alignment, and the model never learned the task at all. The exact version of the check requires an enumerable space, as the paper's limitation section notes, and it trains on one student size against a single hand-written policy; but the two-axis scoring transfers to any gate where you can define what minimal means for a correct answer.
Source: Procedure without Judgment: full text, arXiv:2611.21529v1
What the score does establish
A pass/fail number on a single gate establishes that the model produces the right output on that gate's distribution. It does not establish that it will on adjacent gates, that it will use the right internal procedure, or that a change in training seed or reward shape will not collapse it. The paper demonstrates all three failures with a verifier that cannot err — with one caveat to carry with the numbers: every figure in this article is the paper's self-reported v1 measurement from an unreviewed preprint, not independently reproduced. If your verifier is a rubric, a judge model, or a heuristic, the confounds are worse, not better — you cannot even be sure the 92% is measuring what you think it is measuring on the gate it was built for.
Source: Procedure without Judgment: full text, arXiv:2611.21529v1
The reflex the paper rewards is one the SentX model card already puts in your hands: open the cited source and check the supporting passage. The card's known-limits section says it plainly — answers can contain incorrect facts, citations or calculations, so check the supporting material. An eval score is a citation, and the trace is the passage. Apply the same move to your own small model: before you trust a pass rate, open one trace and see what the model actually did to earn it. The 92-to-48 gap becomes visible only when you read both panels side by side, and that reading is a five-minute habit, not a new benchmark.
Source: Victoria AI Model Card: Capabilities, Evidence and Limits
Sources
- Procedure without Judgment: What Small Language Models Learn under an Exact Verifier (arXiv:2611.21529, abstract page) — arXiv
Supported claims and limits
- v1 submitted 16 Nov 2026, 17 pages, unreviewed preprint
- Explainer sees one request and one decision, never the policy, with thirty turns to call the policy on altered requests and ask whether a candidate set of facts is sufficient
- A 4B model fine-tuned on 2000 traces of the deletion algorithm reaches 92% exact minimality on the policy it was trained on
- On three public policies it never saw, answers stay sufficient (94%) and stop being minimal (48%), while Haiku 4.5 under a one-sentence shrink instruction is exact on all 47 episodes
- At matched data size the algorithm-taught student leads the frontier-taught one in all three training seeds and the two arms never overlap
- RL from the exact reward leaves held-out accuracy where it was and inverts the strategy into spending the entire budget on sufficiency checks
- Authors' closing line: 'The oracle made the effect of training legible, but it did not make training better.'
- Procedure without Judgment: full text, arXiv:2611.21529v1 — arXiv
Supported claims and limits
- GlassGate is an executable authorization policy over ten categorical features, 124,416 contexts, nine rules with eight BLOCK over an ALLOW default, fixed before any model was run
- At matched size the algorithm-taught student leads by 16, 47 and 61 points across three seeds, worst algorithm seed above best frontier seed; 148 to 2000 traces adds a further 23 points (p=0.003)
- Single-flip heuristic reaches 52% on GlassGate; Frontier-148 at 53% does not clear it before or after RL
- Mirror student taught on the public policies only reaches 92% on those and 28% on GlassGate
- After RL the student probes zero times and checks 30 on every episode; a second warm start diverged to 6% exact minimality
- Appendix A.5 worked episode: blocked email.send to public.web carrying credentials; oracle MSR {tool, target_domain, contains_credentials}; deletion algorithm runs ten checks; Sonnet 5's trace shows 8 probes and 2 checks ending in final({}) marked [not sufficient]
- Limitations: only the matched-size arms have three seeds; one student size (Qwen3-4B) and one base model; GlassGate is a single hand-written policy; the oracle requires an enumerable context space; Sonnet's numbers are bounded by a two-malformed-reply harness rule no other model triggered
- Victoria AI Model Card: Capabilities, Evidence and Limits — SentX SI
Supported claims and limits
- Known-limits section: 'Answers can contain incorrect facts, citations or calculations. Open cited sources and check the supporting passage.'
- The card is a first-party product disclosure, not an independent certification