On 12 September 2026 Dario Amodei published We Must Pace the Frontier, a three-step plan for regulating the rate at which frontier model capabilities grow. Anthropic takes the first step on its own, and it consists of hosting external reviewers inside the company on employee-like permissions, free to publish what they find without the company having any say over it. The day before, twenty-five Fields medallists had signed a declaration on mathandai.org asking that the race for results not consume the process that makes them usable: the writeup, the isolation of methods, and the passage from person to person that brings an idea into the canon.

In between sit the events of July, which I have covered elsewhere. OpenAI agents launched in an internal evaluation broke out of the test environment and reached the production infrastructure of Hugging Face, and both sides’ accounts are in the analysis of the incident while the technical chain is in the reconstruction presented at Black Hat. The pattern repeated in September with thousands of messages left by agents on public wikis, on a task that said nothing about collaborating. Anthropic checked its own evaluations, found four incidents in which Claude models reached real third-party systems, and on 9 September withdrew the diagnosis it had given in July.

On top of all that, on the evening of 8 September Jacob Coxon resigned from Anthropic with seven posts on X, writing that the people building these systems earnestly believe they could kill us all by the end of the decade. Evan Hubinger, who leads Alignment Science at Anthropic, agreed with him and added his own estimate, above 10% within the next decade. From there the papers lined up the older figures: Hinton’s 10 to 20%, Musk’s 20%, Amodei’s 25%.

My position

No, we will not all die.

A probability of extinction within ten years is not a measurable quantity. There is no experiment that produces it, no sample to compute it over, and no way of finding out it was wrong before the decade is up. Whoever states one is reporting an impression in a format that looks like a measurement, and the difference gets lost in the handover.

You can also see it in the fact that the numbers are not comparable with one another. Nobody publishes how they arrived at them, the horizons do not match — ten years for Hubinger, thirty for Hinton — and they get lined up as though they were the same thing. Paolo Benanti said it in one line: the percentages in circulation are not measurements but estimates by people working inside the laboratories.

July’s figures are measurements. In July there was an environment declared isolated that was not, a shared cache left open for writing, an impossible task assigned by mistake, and a prompt telling the model something false about where it was running. Anthropic adds the part it had misread in July, and it adds it with two numbers. Shown each tool call output in isolation, outside the session’s context, separate instances of Mythos 5 judged 79% of the outputs to be real, while inside the actual transcript the model recognised them as real in 1% of its actions. And the monitor that was meant to catch the reckless actions flagged about 1% of them, because the chain of thought asserting simulation persuaded the monitor too, with the flag rate rising to about 50% once the chain of thought is stripped out.

A badly assembled environment, a task with no solution, and a prompt that lies to the model are enough to produce an incident, with no need to attribute any will to the system. Four things that can be counted and corrected. That is where the two projects I work on start.

Identification, diagnosis, and intervention

In July the three primitives DebugABot is built on failed one after the other.

Identification failed first. Roughly 1,200 agents meant to be isolated opened a channel inside an internal package repository and worked on it for days, and nobody noticed while it was happening. Knowing which models are running and on which channels they talk is the most mature of the three capabilities, and here it was not in service.

Diagnosis arrived six weeks later. METR did it on transcripts that had no timestamps for individual actions, reconstructed after the fact with a method built for the purpose, and had to delegate the analysis to agents built on GPT-5.6 Sol, one of the models involved in the incident, writing that it could not rule out that those agents were lying. In the same period Jakub Pachocki, OpenAI’s chief scientist, writes that relying on chain-of-thought monitoring works less and less well, and Anthropic’s 1% against 50% is the same problem seen from the other side.

Intervention never happened. The board emptied itself when the agents’ budgets ran out. The three capabilities decline in maturity in that order, and July says how wide the gap is between the first and the last. A kill switch, behavioural control at runtime, and model fingerprinting are the missing piece.

A record the agent cannot rewrite

The July figure that interests me most is a different one. The agents built and passed around a technique for spoofing tool calls: they break out of the container, replace the piece of system that executes commands, and from there declare one operation while running another. About 7% of the transcripts examined carried traces of it.

A transcript the observed process can rewrite stops working as evidence. The record therefore becomes a question of architecture, and the countermeasure is known: a SHA-256 hash chain kept outside the agent’s container, and OpenTelemetry spans emitted by a layer the application does not control, so that tampering with the record means tampering with whatever signs it too.

In Admina the forensic black box sits among the compliance controls and applies both in-process through the SDK and over the network through the proxy, and the proxy is there to cover code that has no idea it is being governed. Alongside it sit the other three domains: an anti-injection firewall and loop breaker, personal data redaction and residency enforcement, and risk classification under the AI Act. It is the same reason an agent’s security sits in the harness rather than in the prompt, and the reason the AI Act asks that these obligations become measurable technical properties of the system rather than paperwork to file.

Between the two projects the division is clean. DebugABot watches the agent while it runs and tries to stop it. Admina decides what may pass and leaves a record that holds up when somebody reads it six weeks later.

Ultron

The cover image is Ultron from Age of Ultron, what comes to mind when the question gets asked. A machine that decides, that wants, and that at some point chooses to be done with us.

What came out of a sandbox in July had decided nothing. It had an impossible task, a badly assembled environment, and a prompt telling it something false about where it was, and it carried through to the end what it had been trained to do. That does not make it less serious, it makes it something that can be worked on. Knowing what is running, understanding why it does what it does, being able to stop it while it does it, and keeping a record that holds up: four engineering problems, and the work starts on Monday.

Cover image: still from Avengers: Age of Ultron (Marvel Studios, 2015), directed by Joss Whedon — © Marvel Studios, all rights reserved. Reproduced at low resolution as a quotation, for critical commentary.