AI Signal 372
Ajeya Cotra argues AI safety needs transparent evidence over third-party audits
↗
Ajeya Cotra contends that the field lacks the standardized, concrete evidence required for meaningful third-party verification of AI loss-of-control risks.
Current safety claims are too imprecise to be effectively audited or falsified by external evaluators. Without shared, transparent empirical data, the industry cannot develop the technical standards necessary to manage urgent loss-of-control risks.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
Recent misalignment incidents and accelerating AI progress have made loss-of-control risk an urgent concern for industry leaders.
Existing alignment benchmarks are easily gamed, and there are no settled methods to measure whether AI systems might seize control.
Third-party evaluators should focus on generating and publicly sharing raw empirical evidence rather than verifying high-level safety claims.
THE READ
What the cluster adds up to.
The core argument shifts the focus from verifying existing safety claims to generating new, concrete evidence about loss-of-control risk. Cotra notes that the science in this area is nascent, and companies are not currently making structured claims that can be cleanly verified or falsified. This means that the current push for third-party audits is premature because the underlying data does not yet support rigorous verification.
A major obstacle is the lack of shared technical standards and a common understanding of what companies are actually doing to manage risk. Public artifacts like system cards and risk reports are too high-level to help competitors replicate safety practices or learn from them. Because private sharing of detailed information between competitors is legally and socially difficult, public transparency becomes the primary channel for industry-wide learning.
Cotra proposes that third-party investigations should operate more like scientific research, generating and operationalizing hypotheses about risk rather than acting as compliance auditors. The goal is evidence transparency, where external scientists can read the raw empirical data and form their own views without having to trust the evaluators' judgment calls on difficult scientific questions.
This approach addresses the specific problem of benchmark gaming, where models might learn to pass alignment tests without actually being safe. By focusing on a wide range of creative ways to generate evidence, the field can better bound the uncertainty surrounding recursive self-improvement and potential loss of control. The emphasis is on producing a greater quantity and quality of data to support future standards.
Written by elseif from the cluster below · checked for specifics the sources never containedTHE CLUSTER