Apollo Research wants AI safety claims that outsiders can test

The short version

Apollo Research's latest proposals ask a useful question about AI assurance: if a developer's safety claim were false, would an outside reviewer have a fair chance of discovering that?

Its September 30 principles propose reviews built around specific claims, evidence and public findings. Its October 1 scheming proposal applies that approach to models covertly pursuing goals their developers did not intend. Both pieces were linked in Apollo's captured X intake and checked on its official site.

These are proposals, not proof that a new review regime has reduced risk. Their practical value is a way to ask more precise questions of an AI supplier, and to make your own release claims easier to examine.

Core ideas

An embedded evaluator is an outside reviewer with continuing access inside the developer's process. Apollo argues that this reviewer needs meaningful independence, sufficient access and a route to publish findings. A brief look at selected outputs would provide weaker evidence.

The principles distinguish several outcomes. A claim can remain unrefuted even when the methods are too weak to support much confidence. A developer can report contrary evidence itself. An evaluator can uncover it. Access can be blocked. Even the strongest proposed verdict retains uncertainty.

For an ordinary product team, our inference is modest: the evidence should say what was examined and what remained unseen. A label on a slide cannot carry all of that detail. A reviewer who could not inspect an important component should record the gap rather than treating it as a clean result.

Consider a hypothetical assistant that changes customer subscriptions. A team might say that it never changes a plan without approval. That claim becomes useful only when the test covers the real action path and the reviewer can inspect what counted as approval. A demo with a friendly request would leave much of the claim untested.

What they've been saying lately

The October 1 piece separates four claims: training did not encourage scheming; strong tests found no tendency to scheme; internal use showed no attempts; and monitoring would detect scheming reasoning. Apollo argues that assessing these requires sustained access to training and internal activity, rather than only testing a finished model. It calls the claims necessary but insufficient for a complete safety case.

The two posts mark a move, within this recent pair of publications, from review principles to a specific problem those principles should address. We are not claiming a complete history of Apollo's work. The first piece asks how assessment should operate. The second asks what a reviewer needs to establish about covert behavior.

Our practical reading is that teams should separate different kinds of evidence. A test suite examines selected situations. An activity log records what happened in a particular environment. A monitor test checks whether the system would notice a known problem. None automatically substitutes for the others.

That separation can prevent a familiar release-review mistake: using a result from one setting to answer a broader question about another. A support assistant that passed a sandbox test has evidence about that test. It still needs a review of the permissions and monitoring used in production.

Where they sit in the debate

Apollo's stance emphasizes access and the ability to challenge a claim. Its proposed reviewer does more than accept a developer's test report. The reviewer examines evidence and looks for reasons the claim might fail.

This article does not establish whether every developer would grant the required access, or whether evaluators could scale that work fast enough. Those are open implementation questions. Nor do these posts establish that the proposed labels will be interpreted consistently across reviewers.

Our concern for buyers is more immediate. Different suppliers may use the word evaluation for very different work. Before comparing assurances, ask whether the review covered the same tasks, tools, versions and operating conditions. A comparison can otherwise reward the broadest wording rather than the strongest evidence.

Teams should also avoid turning a finding into an unlimited promise. If a test covers one release and one set of permissions, say so. A later model, new tool or changed account can require a fresh decision. This is our recommendation for operational use, not a compliance requirement created by Apollo's posts.

Start here

Read the principles first to understand the proposed relationship between developer and reviewer. Then read the scheming piece as a worked application. Keep the distinction between a proposal and a measured finding visible throughout.

We recommend borrowing one small practice for your next release: write a claim that could be disproved. Instead of saying an assistant is safe, state a bounded behavior such as no subscription change reaches the billing service without recorded approval.

Ask what evidence could contradict it, who can inspect that evidence and what the test cannot see. Keep the unresolved questions in the release record. This produces something a colleague can review without adopting Apollo's entire frontier-model framework.

What this means for your work

For business and technology leaders

When buying an AI support tool, ask what the outside reviewer could actually inspect. For example, a supplier may show a clean test report while leaving the billing integration outside its scope. We recommend requesting the test scope, access limits and unresolved findings before expanding the pilot. This week, choose one supplier claim and ask which evidence would show it was wrong. Use the answer to decide whether you need more review time or narrower access.

For full-stack developers

Make one release claim testable through the real service path. For a subscription assistant, record the request, approval and accepted billing action together. We recommend adding a test that tries to change a plan without valid approval and checks that the service denies it. Then document which paths and settings the test covers. That gives reviewers a specific result to examine while keeping the remaining uncertainty clear.

Sources

• Principles for Embedded Evaluations — Apollo Research, official proposal, September 30, 2026; distinct piece linked in captured X intake.
• Towards embedded evaluations for scheming propensities — Apollo Research, official proposal, October 1, 2026; distinct piece linked in captured X intake.