Automated Proof: Anthropic's Claude Generates Software Safety Certificates

Anthropic's AI assistant Claude has produced roughly 180 million lines of proof code in just a few weeks to mathematically verify the safety of existing software. The effort, conducted in partnership with a firm called Theorem, was designed to demonstrate that machine-generated code can be proven correct. This approach attempts to solve a significant problem regarding the reliability of AI programming by providing mathematical guarantees rather than relying solely on traditional testing methods.

Traditionally, proving that software runs error-free under all circumstances has been prohibitively expensive and time-consuming. For example, verifying a small operating system kernel previously required a team of specialists spending 20 person-years on the task. Because of these high costs, the method has generally been reserved for extreme cases, such as avionics software used by Airbus. However, the Frontier Red Team at Anthropic utilized Claude to automate this proof generation process.

Instead of creating new software, Claude was tasked with verifying existing code by translating crash tests into mathematical problems. The system analyzed compiled machine code to produce extensive proofs for 123 open-source libraries. These libraries are part of Google's OSS-Fuzz security project. An independent verification program called Rocq automatically checks the proofs generated by the model, a process the researchers describe as fully autonomous.

During this verification process, Claude identified hundreds of bugs that conventional testing had missed. Many of these issues only appeared under very specific conditions, such as unusually large input files or particular sequences of data access. For instance, while verifying a widely used PDF reader, the team found as many security bugs in one week as are typically reported for that software in an entire year. In another application, Claude caught a logical error that had gone undetected for over a decade. Anthropic states that even when a complete proof fails, the attempt still surfaces real bugs.

Anthropic acknowledges that being formally verified does not mean the software is entirely error-free. The proofs hold only under strict conditions, such as using a specific processor architecture, single-threaded programs, and inputs under 64 MB. Additionally, the compute cost is high, with individual proofs running up hundreds of millions of tokens, though total project costs were not reported. However, the company notes that if AI can verify its own code, developers can trust AI-generated software more and spend less time reviewing it.

The project strengthens the case for software automation. If AI can write code and guarantee its correctness under defined conditions, the question shifts from whether AI can program to whether it can prove its code is safe. Anthropic expects formal verification with AI to soon become as practical as standard testing. The roadmap for Theorem already targets critical infrastructure like glibc, OpenSSL, and cURL.