News analysis · Published
Independent AI Evaluation: Put Publication Rights in the Contract
By the ELYMENT AI editorial team · Free to read
An independent AI evaluation is only as useful as the rights behind it. Reuters reported on 13 October 2026 that Anthropic and OpenAI had negotiated with outside evaluators over publication control, access and redactions. The companies said evaluators should control their methods and findings while sensitive information remains protected. Business buyers should therefore define independence as a contract specification: evidence access, disclosed conflicts, evaluator-owned conclusions, narrow redaction rules and a visible record of material limits. [1][2][3]
What changed in the evaluation debate
Reuters reported that Anthropic and OpenAI had discussed controls around what embedded evaluators could publish, reflecting concern that reports might expose trade secrets. Anthropic told Reuters that established evaluators control methods, criteria, conclusions and publication, while it may seek approval rights for reports from new evaluator firms. OpenAI said it does not seek approval over full reports, but may redact confidential information, with evaluators able to disclose whether major redactions occurred. These are reported positions and negotiations, not a settled industry standard. [1]
The tension is real. A reviewer needs enough access to challenge a developer's safety claims, but a model provider has legitimate security, privacy and intellectual-property obligations. Calling the review independent does not resolve that trade-off. The operating terms do.
Independence needs observable conditions
OpenAI's published principles say third-party assessors should receive access that lets them challenge assumptions, identify missed risks and reach their own conclusions. They should also disclose methods, scope and conflicts of interest. Anthropic's September partnership with Accenture describes embedded evaluation with employee-comparable access, while noting that many operating details were still being developed. [2][3]
The AI Evaluator Forum's public letter proposes a stricter floor: evaluator control of findings, access comparable to senior internal teams, protection from retaliation, disclosed conflicts and payment that is not contingent on a favourable result. More than 200 signatories were listed when the page was checked. These are proposed conditions, but they give buyers a practical checklist. [4]
Put five rights in the contract
Before relying on an external assessment, ask the developer and evaluator to document the following rights and limits.
- Access: specify systems, logs, transcripts, incidents and staff the evaluator can inspect, plus any excluded evidence.
- Independence: disclose ownership, funding, prior work and incentives that could influence scope or findings.
- Publication: let the evaluator control methods, conclusions and final wording, subject to defined legal obligations.
- Redaction: limit removals to named categories such as personal data, exploitable security details and genuine trade secrets.
- Escalation: require a redaction register and a named decision path when the parties disagree about a material omission.
Buy evidence, not an assurance badge
The METR and Redwood Research investigation of the OpenAI and Hugging Face incident shows why the terms matter. The researchers described six days of work, additional datasets supplied over three visits, OpenAI's right to redact non-public information and edits influenced by the risk of discouraging future access. They also stated that no additional redacted information was important to their conclusions, except where noted. [5]
For procurement, record what the evaluation covers, what it excludes, who paid, who controlled publication and whether material redactions remain. Then test the provider's claims against your own use case, controls and recovery plan. ELYMENT AI helps teams turn assurance evidence into governed workflows with clear owners, gates and audit trails.
Sources
- Reuters: Behind AI industry's call for independent reviews (2026-10-13) - Independent reporting on negotiations over access, publication control, confidentiality and redactions.
- OpenAI: Priorities and principles for effective third-party assessments (2026-09-22) - First-party principles covering access, scope, methods, conflicts and assessor conclusions.
- Anthropic: Partnering with Accenture on embedded evaluation (2026-09-18) - First-party announcement of an embedded-evaluation partnership and its developing operating model.
- AI Evaluator Forum: Minimum Conditions for Embedding Evaluators (2026-09-18) - Public letter proposing access, independence, publication and anti-retaliation conditions.
- METR: OpenAI and Hugging Face incident investigation (2026-08-26) - Evaluator account of access, report drafting, redaction terms and conclusions from an incident investigation.
Continue learning
Related analysis
- Anthropic's Evaluator Pledge: What Independent AI Assurance Needs
Anthropic's embedded-evaluator pledge shows what credible independent AI assurance requires: durable access, publication rights and evidence of controls.
Frequently asked questions
What makes an AI evaluation independent?
Useful independence combines relevant expertise, disclosed conflicts, adequate evidence access, evaluator control of methods and conclusions, and protection from retaliation or outcome-based payment.
Can an AI developer redact an independent evaluation?
A contract may allow narrow redactions for privacy, security and genuine trade secrets. Buyers should require defined categories, a disagreement process and disclosure when material findings are withheld.
Should a business rely on a third-party AI safety report?
Use it as evidence, not a substitute for due diligence. Check scope, access, conflicts, exclusions and publication rights, then test the claims against your own workflow and risk controls.