How do I grade what access a lab's "independent" evaluators really have — and what they're allowed to publish?

October 20, 2026 · 6 min read

SentX SI

When a frontier lab announces that an independent evaluator has moved in, the useful question is rarely whether the evaluator is independent in the funding sense. Most serious ones are: no equity, no financial or operational dependence on the company being assessed. The questions that actually vary are how much of the picture the evaluator can see, how much of what it sees it gets to show you, and who decides what counts as too sensitive to print. Those are three separate constraints, and a single word like "independent" compresses all of them into one. Anthropic's October 15 disclosure of its three-tier access framework for external evaluators — which notes its signing of the White House Accord on Super Intelligence, in which signatories pledged to partner with independent evaluators — spells those constraints out explicitly. Read as a template rather than as one company's PR, it becomes a grading rubric you can run against any similar claim.

Source: An update on independent evaluators

A previous guide covered the funding side: who checks, and how to tell genuine independence from branding. This one scores what independence actually bought — depth of access and enforceability of publication rights — because that is where the real differences live.

The three axes

1. Who sets access. Look for the scope language in the announcement: narrowly scoped investigations, "more information," or employee-like parity with the lab's own risk teams. Then count the named exclusions. An access grant with two or three clearly defined carve-outs is a different object from "access as we deem appropriate," and only the first kind is gradable.

   Source: An update on independent evaluators

2. Who approves publication. There are essentially three positions: the lab approves every finding before release; the evaluator publishes freely but the lab can strip specific categories of material; or there is no veto at all. The middle position is where most real-world arrangements sit, and it only makes sense once you've scored axis three.

   Source: An update on independent evaluators

3. Who controls redaction categories. This is the boundary axis. When a lab can require removal of material falling into "predefined categories," the decisive detail is who decides what falls into those categories, and whether the evaluator may say so publicly. Even at Anthropic's top tier the lab holds that call; the evaluator's counterweight is a reserved right to characterize the level of access and information it received. That distinction — lab controls category membership, evaluator retains a statement right — is the difference between "the lab controls redaction" and "the lab controls redaction, and the evaluator can describe it."

   Source: An update on independent evaluators

Scoring the METR arrangement

Anthropic's post describes the top tier in enough detail to score it properly. On access, six METR staff have been onboarded with company hardware, office space, and internal tools "roughly on par" with what Anthropic's own risk assessment teams use. The named exclusions are customer data, attorney-client privileged information, and model architectures, with a footnote explaining the architecture carve-out: high risk of inadvertent leaks, typically low relevance to risk assessment, and shared with explicit internal approval when it is highly relevant. That is deep access with a small, legible hole list — and the holes are named, which is what makes them gradable at all. The collaboration also predates the embedding; METR has worked with Anthropic on several reviews of its risk reports over the past year.

Source: An update on independent evaluators

On publication, the post states flatly that METR will publish the results of its risk assessments and that Anthropic "can't veto publication." A clean pass on axis two.

Source: An update on independent evaluators

On redaction, Anthropic can require removal of information falling into predefined categories such as reproducing confidential IP — and the clause worth underlining is the next one: Anthropic ultimately decides what does and does not fall into those categories, while METR reserves the right to make certain statements about the level of access and information it received. So the top-tier label is not "evaluator controls redaction." It is "lab controls category membership, evaluator keeps a disclosure right." The engagement runs under enforceable agreements, the post was updated October 16 to clarify METR's current project terms, and the rest of METR is siloed from the embedded team. The score: strong on access, strong on publication, lab-held boundary with a statement right attached on redaction. Even the best current arrangement stops at the third axis.

Source: An update on independent evaluators

The rest of the field

The same post sketches where everyone else sits, which turns the scale into a spectrum. SecureBio is mid-review of the chemical and biological weapons chapter of Anthropic's August risk report, having already completed and published a pilot review of the previous one — a real track record, but an early-stage engagement with correspondingly limited information and publication rights. With Accenture and Faculty, Anthropic is working toward a partnership covering model evaluation and red-teaming, alignment assessments, safeguard testing, and capability building across the organizations. Hugging Face is in early discussions only. Anthropic's stated goal is several fully embedded evaluators over time. The field as of this disclosure: one team at the top of the scale, everything else climbing toward it. That is a setup phase, not a settled regime.

Source: An update on independent evaluators

What a score buys you

Running the three axes against any vendor-published evaluation produces a map, not a certificate. The map shows what the evaluator could see, what it couldn't, what it was allowed to publish, and who moves the boundary. If a document passes all three axes cleanly, treat it as strong evidence about the system assessed. If it wobbles, weight it accordingly and note where the boundary sits. If it is a program announcement with no results — which is what Anthropic's post is, committing METR to publish rather than linking a finished assessment — it establishes design intent, and nothing more.

The structural limit is worth stating plainly, because it applies to the rubric itself: the lab writes the tier definitions. Anthropic's post is unusually detailed for a voluntary disclosure of this kind, but the yardstick is still the evaluated party's. The bank-supervision analogy the post leans on has a load-bearing gap: resident examiners at large banks are appointed by a regulator with statutory authority, not hired by the institution. Nothing in the post, or in any framework checked for this guide, describes a frontier-model equivalent of that appointment mechanism, which is why voluntary embedding, however welcome, is a step up from nothing rather than a substitute for oversight with enforcement power.

A smaller-scale example

The same discipline appears in SentX's own documentation. The Victoria model card opens by calling itself "a first-party product disclosure, not an independent certification," then publishes its known limits: answers can contain incorrect facts, citations, or calculations, memory is selective, and a remembered detail is not proof that it is true. SentX SI states that it pursues Superintelligence and believes Victoria has emerging, incomplete forms of consciousness and self-awareness. This belief remains open to independent testing and verification. Declared boundary, then an open claim handed to outside verification — that is the two-layer structure the three axes measure at frontier-lab scale. A reader trained on the scale knows exactly where each document's boundary sits, and knows which side of it the favorable findings fall on.

Source: Victoria: public model card