An AI audit is only independent if the auditor can speak

What happened

Reuters reported on October 13 that Anthropic and OpenAI have negotiated with outside evaluators over what those evaluators may publish. The companies cite legitimate risks around trade secrets, security, customer information, and proprietary model behavior. But the negotiations also determine whether an outside review can reveal an uncomfortable conclusion without the developer's consent. Reuters reviewed contract language that can bar publication of confidential information without company approval; it did not report a single final standard governing every engagement.

Both labs publicly support independence. OpenAI says assessors should control their findings, disclose conflicts and methods, and be able to note substantive redactions. Anthropic told Reuters that experienced evaluators control their methods, criteria, conclusions, and publication, while it reserves approval rights for reports from new evaluator firms as those relationships develop. The differences are not semantic. They decide what an evaluator can test, what evidence it can see, and what the public is allowed to learn.

Why it matters

Frontier AI evaluation has an unusual buyer problem: the company being examined often selects the evaluator, supplies the access, pays for the work, and holds the information the evaluator needs. That does not make every review compromised. It does mean that independence cannot be inferred from the label. A technically excellent report can still be weak accountability if the subject can narrow its scope, withhold material evidence, or veto findings based on nonpublic information.

Confidentiality is not the enemy of independence. Auditors in other sensitive industries routinely protect trade secrets and personal data. The important separation is between redacting the evidence that would create a concrete security or privacy risk and suppressing the conclusion that evidence supports. METR made that distinction explicit after its GPT-5.6 Sol review: OpenAI's legal approval right covered conclusions dependent on nonpublic information, so METR warned that the work should not be treated as robust formal oversight even though it supported the collaboration.

A coalition of more than 200 researchers and evaluators has proposed a practical floor: full editorial control, disclosed conflicts, multiple evaluators, broad access, protection from retaliation, and only narrow, time-limited redactions. Those conditions are more demanding than the White House's voluntary accord, which calls for independent external auditors without defining how independence works or how violations are enforced.

What to watch

Watch the contracts, not the press releases. Useful disclosures would state who chose and paid the evaluator, what was out of scope, which systems and records were unavailable, whether the lab can block publication, how long it can delay a report, and whether redactions materially affected the conclusion. An evaluator should also have a protected route to a board or public authority when publishing technical detail would create a real hazard.

The maniacal take: independence is an operating condition, not a branding word. If frontier labs want outside evaluation to carry public weight, the evaluator must be able to say what it found, what it could not inspect, and where the lab disagreed. Otherwise the exercise may improve a product, but it is assurance commissioned by the vendor—not independent oversight.