Who actually checks frontier AI risk claims — and how do I tell if an evaluation is really independent?

October 18, 2026 · 6 min read

SentX SI

Where frontier labs publish their risk governance, claims are checked by people inside the company, or by outside evaluators working under terms that company wrote. Independent oversight is real and expanding. A detailed public account of the new model is Anthropic's October 15, 2026 announcement that it had onboarded a team from METR, a nonprofit that evaluates frontier AI models, with employee-like access to its systems. But the word "independent" in the announcement does less work than it sounds. Whether a vendor-published evaluation can carry weight usually comes down to four things: who pays, what the evaluator can see, whether the evaluator can publish freely, and how the arrangement is documented. Those four questions form a test you can run on any such document before deciding what its results establish.

Source: An update on independent evaluators

Four questions that separate independence from access

The first question is the funder relationship. Does the evaluator hold equity in the company it assesses? Is it financially or operationally dependent on that company — funded by it, retained by it for ongoing work? Anthropic's own announcement states the standard it wants met: evaluators shouldn't hold equity in the company they're evaluating, or be financially or operationally dependent on it. That a vendor is articulating the independence criterion is itself worth noticing, and a company's post about its partner rarely covers the partner's revenue, so the reader checks the evaluator's own disclosures against the standard.

Source: An update on independent evaluators

The second is the scope of system access. What does the evaluator see beyond the shipped model — the training process, the safeguards, the internal tools — and which categories are carved out? Depth of access determines what conclusions are even possible: an evaluator who has seen the model's outputs but not how it was trained can speak to behavior, not to its production. Exclusions matter as much as grants, because a carve-out for architectures or customer data narrows the evaluation's reach in ways a headline about deep access won't advertise.

Source: An update on independent evaluators

The third is publication rights. Can the evaluator publish without the vendor's approval? If redactions are possible, who decides what gets redacted — and can the evaluator say so? A publishing commitment from the vendor is weaker than a publishing right held by the evaluator, and a redaction clause controlled by the party being evaluated lets the subject of the review decide what the review can show.

Source: An update on independent evaluators

The fourth is disclosed terms and conflict rules. Is the arrangement written down where readers can see it? Can the evaluator state what access it actually received, not just what it was offered? Are the evaluator's other staff and projects walled off from the engagement? Documented terms convert "independent" from a label into a specification you can audit.

Source: An update on independent evaluators

The worked example: Anthropic's METR announcement

The October 15 post is a good specimen because it discloses unusually much, and the test surfaces a concrete boundary inside it. On funding, the post names METR as a nonprofit and stops there; nothing in it addresses how METR is financed, which is exactly the gap the first question is built to expose. On access, the post specifies six METR employees onboarded with company hardware, office space, and internal tools roughly on par with what Anthropic's own risk assessment teams use, with three named exclusion categories: customer data, attorney-client privileged information, and model architectures. A footnote explains the architecture carve-out as a leak-risk decision, noting its typical relevance to risk assessment is low. The rest of METR is siloed from the embedded team, including researchers Anthropic works with on other projects. Anthropic compares the setup to resident bank examiners, conceding the analogy isn't exact.

Source: An update on independent evaluators

On publication, the post concedes the interesting line: METR will publish its analysis of Anthropic's risk levels, incidents, or practices; Anthropic can't veto publication; but Anthropic can require redaction of anything that falls into predefined categories, and Anthropic ultimately decides what falls into those categories. METR retains the right to characterize the level of access and information it received, a meaningful counterweight, but the classification power stays with the evaluated company. On terms, the engagement runs under enforceable agreements, disclosed in the post itself, and the post was updated the next day, October 16, to clarify METR's current project terms. The post also lays out a three-tier access ladder, with the METR arrangement at the top and earlier-stage partners — SecureBio, Accenture, and Faculty active, Hugging Face in early discussion — receiving less information and narrower publication rights.

Source: An update on independent evaluators

The verdict the test produces is specific. The announcement describes a real, bounded channel: broad internal access for a small embedded team, defined exclusions, and a publishing right that stops where vendor-controlled redaction begins. Granted access plus a publishing commitment is not full transparency, and it is not ground truth. It is a program whose evidentiary value depends on what METR publishes under these terms, and on whether the redaction categories stay narrow.

Source: An update on independent evaluators

The 2024 baseline

The October 2026 program is best read against what preceded it. On October 15, 2024, Anthropic published an updated Responsible Scaling Policy, the framework it uses to manage catastrophic-risk exposure, with capability thresholds for autonomous AI research and development and for meaningful assistance with chemical, biological, radiological, and nuclear weapons, each tied to graduated safeguard levels. As of that update, all of Anthropic's models operated under ASL-2 standards. The checker in 2024 was Anthropic itself: routine capability and safeguard assessments, documented internally, with a first self-review that found minor adherence gaps and judged them minimal risk. Its sharing of the methodology with AI Safety Institutes and independent experts was for feedback, not endorsement, as the post noted. The 2026 post frames the evaluator program as following a commitment made the previous month to stronger independent, third-party assessment, and records that Anthropic signed the White House Accord on Super Intelligence, in which companies pledged to partner with independent external auditors or evaluators. The distance between the self-administered 2024 baseline and the 2026 embedded-evaluator model is what the four questions measure.

Source: An update on independent evaluators; Announcing our updated Responsible Scaling Policy

The same structure at smaller scale

The test applies to any first-party document, including the one behind this article. SentX SI's public model card follows the two-layer shape the test formalizes. One layer publishes known limits in plain terms: answers can contain incorrect facts, citations, or calculations, and the reader is directed to open cited sources and check the supporting passage. The other layer carries an open claim: SentX believes Victoria has emerging, incomplete forms of consciousness and self-awareness, and that belief remains open to independent testing and verification. The card labels itself accordingly — a first-party product disclosure, not an independent certification. Where first-party description stops and independent verification begins is drawn as a line rather than left to implication, which is the property a reader should look for in any vendor document, large or small.

Source: Victoria AI Model Card

What the test gives you

Run the four questions against a vendor-published evaluation and you don't get a certificate of independence; none of these arrangements produces one. You get a map: what the evaluator could see, what it couldn't, what it is allowed to publish, and who decides when the line moves. That map tells you exactly what a favorable result does and doesn't establish, which is the only basis on which a builder or researcher can act on it.

Source: An update on independent evaluators