In March 2023, more than thirty thousand people — including Yoshua Bengio, Stuart Russell, Elon Musk, and Steve Wozniak — signed an open letter calling for every AI lab to pause, for at least six months, the training of systems more powerful than GPT-4. It was serious, well-argued, and signed by some of the most credible names in the field.

Not a single lab paused. Two and a half years later, GPT-5 shipped.

The letter’s failure is easy to wave away as naïveté. But the more telling event came in February 2026, and it came from the one company you’d least expect. Anthropic — the lab founded by people who left OpenAI specifically over safety, the lab that built its entire identity around responsible scaling — quietly removed the clause in its own policy that committed it to pausing development if its models outran its ability to control them.

The most safety-committed frontier lab on earth dropped its own pause commitment. And its stated reason is the whole story: if it slowed down while others didn’t, it would simply lose, and the least careful developers would set the pace.

That’s not a betrayal of principle. It’s the principle colliding with game theory. A pause requires everyone to stop at once, and nothing in the current structure makes that possible.

For the rest of us: why “just pause” is harder than it sounds

The intuitive case for a pause is simple: this is powerful, possibly dangerous technology, we don’t fully understand it, so let’s stop and make sure it’s safe before going further. Sensible. The problem isn’t the logic. It’s the coordination.

Imagine several runners in a race where stopping to tie your shoes properly is the safe, responsible thing to do — but whoever stops gets overtaken and may never catch up. Every runner agrees, in principle, that everyone should stop. But no individual runner can afford to stop unless all the others do at exactly the same time, and there’s no referee who can make them. So everyone keeps running, each privately wishing the whole thing would slow down.

Economists call this a collective-action problem, or in its sharpest form a prisoner’s dilemma: the choice that’s best for the group (everyone pauses) is not the choice that’s safe for any individual (pause and fall behind). Without an enforcement mechanism that binds everyone simultaneously, the rational move for each player is to keep going — even when they’d all prefer the world where everyone stopped.

That’s the AI race in one paragraph. And it’s why “trust me, I’ll stop” is worth exactly nothing without a way to make everyone else stop too.

The 2023 letter: 30,000 signatures, zero pauses

The Future of Life Institute’s open letter was not a fringe document. It gathered tens of thousands of signatures, including pioneers of the field. It named concrete risks. It asked for a modest, time-boxed six months.

The result was a clean natural experiment in whether moral suasion can halt an arms race. The answer was no. Development didn’t slow; if anything it accelerated, as capital and talent poured in. The letter did shift the conversation — it helped legitimise public anxiety, nudged regulators, and put safety on the agenda. But on its literal ask, the pause, it achieved nothing, because it had no mechanism to make stopping safe for any individual lab.

A request to pause, with no way to bind everyone, is just a request to fall behind.

Anthropic’s reversal is the real proof

If you wanted to design an experiment to test whether any lab could unilaterally hold the line on safety, you’d pick Anthropic. It was founded by former OpenAI researchers who left over safety disagreements. Its public theory of change, “race to the top,” is about setting an example others feel pressure to follow. Its Responsible Scaling Policy was the gold standard for self-binding commitments — and it explicitly promised to pause development and deployment if capabilities outran the company’s ability to mitigate the risks.

In RSP version 3.0, effective 24 February 2026, that pause commitment is gone.

Anthropic’s own explanation is remarkably candid. The level of catastrophic risk, it argues, depends on what all frontier developers do. If Anthropic paused to implement safety measures while others pressed on, the result could be “a world that is less safe” — because the developers with the weakest protections would set the pace, and the responsible ones would lose the ability to do safety research or stay relevant at all.

Read that carefully, because it’s the FLI letter’s failure restated from the inside. The most safety-motivated lab in the industry concluded that unilateral restraint is not just costly but possibly counterproductive. When the strongest believer in pausing decides it can’t pause, the debate about whether a voluntary pause is feasible is over. It isn’t.

Why the atomic-bomb comparison both fits and fails

The natural analogy is the bomb. “Let’s all stop building nuclear weapons — trust me, I’ll stop first” was never going to work either, for exactly the same arms-race reason. The Manhattan Project ran because the alternative was assumed to be someone worse getting there first. The logic is identical.

But the analogy breaks in the place that matters most. Nuclear weapons were built by states, behind classification, with a containment doctrine baked in from the start — and after the fact, a handful of governments could police the rest by treaty, because the inputs (enriched uranium, test sites) were physical, scarce, and trackable.

Frontier AI has none of that. It’s built by private companies across multiple jurisdictions, the compute is portable, the weights are copyable, and once a capable open model is released it can’t be recalled. There is no Atomic Energy Commission for AI, no treaty body, no equivalent of fissile-material accounting. (I wrote about this structural difference in The First Dangerous Technology Built Without the State.) So even the imperfect coordination that eventually constrained nuclear weapons is far harder to reproduce here. The arms-race dynamic is the same; the tools that once contained it are mostly absent.

What this means

If you’re waiting for the industry to hit pause before you think seriously about AI governance, stop waiting. The pause isn’t coming. Not because the people in the field are reckless — many of them genuinely want it — but because the structure makes unilateral restraint a losing move, and there’s no mechanism to make restraint collective.

That has two practical consequences.

Governance has to come from binding, multilateral mechanisms, not voluntary promises. A self-imposed commitment lasts exactly until it becomes a competitive disadvantage — Anthropic just demonstrated the half-life. Anything that actually slows the race has to bind everyone at once, which means treaties, enforceable regulation, or hardware-level controls, not pledges.

For everyone downstream, assume the frontier keeps moving. If you deploy AI in an enterprise, build your own controls, your own review layers, your own governance — because the external brakes you might be counting on don’t exist and aren’t being built fast enough. The people closest to the technology are, for now, the only governance layer there is.

The honest summary is uncomfortable: the smartest, most safety-conscious people in the field have looked at the arms race, agreed it’s dangerous, and concluded individually that they cannot afford to be the one who stops. That’s not a moral failure. It’s a coordination failure. And coordination failures are not solved by asking nicely.

References

  • Future of Life Institute, Pause Giant AI Experiments: An Open Letter (March 2023) — 30,000+ signatories incl. Bengio, Russell, Musk, Wozniak, Harari; called for a 6-month pause on training systems more powerful than GPT-4.
  • MIT Technology Review, “What’s changed since the ‘pause AI’ letter” (September 2023); Axios, “No one took a six-month pause” (September 2023).
  • Anthropic, Responsible Scaling Policy v3.0 (effective 24 February 2026) — removes the prior pause commitment; cites the collective-action problem. Coverage: CNN Business, Transformer, GovAI analysis (February 2026).
  • Anthropic, “Race to the Top” theory of change (Responsible Scaling Policy materials, 2023–2026).
  • keller-ai — The First Dangerous Technology Built Without the State.