The company behind Claude warns that artificial intelligence is approaching a dangerous threshold — one where systems can rewrite their own code, conduct research, and potentially build themselves without human oversight. The call for a moratorium is as controversial as it is unprecedented.
What to know
- Anthropic has publicly urged top AI labs to slow down development, warning that self-improving AI could emerge without human input.
- The company’s CEO stated that AI now writes most of its code and runs increasingly complex research tasks; humans are becoming the bottleneck in deciding which problems are worth solving.
- A leaked report from Anthropic suggested that within the next few years, AI systems could build and improve themselves with minimal human involvement.
- The call is complicated by revelations that Anthropic embedded engineers at the NSA for offensive cyber operations — a move that contradicts its public safety advocacy.
- Critics argue that a global development pause would solidify the dominance of current leaders like Anthropic, creating a de facto monopoly in frontier AI.
- The debate highlights deep tensions within the AI community between accelerating capabilities and ensuring long-term safety.
- Regulators and lawmakers are now watching closely, with potential policy shifts on the horizon.
The Call for a Pause
In early June 2026, Anthropic made a striking plea to the global AI community: halt the training of systems more powerful than current frontier models. The proposal, published on the company’s official channels, argues that the industry is approaching a inflection point where AI could begin to self-improve autonomously. Without a coordinated pause, the risk of losing human control becomes unacceptable.
The timing is notable. Just days before, the same company reported that its own AI now writes the vast majority of its code. Engineers, the report explains, spend their time not on coding but on deciding which research problems to pursue. The implication is clear: the line between tool and creator is blurring.
Anthropic’s leadership emphasizes that this isn’t about halting progress entirely — only the most dangerous races. But the proposal’s boundaries are vague, and enforcement mechanisms remain unspecified.
Code Writing Itself
The idea of AI writing its own code is not new, but Anthropic’s admission that it now happens at scale inside the company’s own labs is a watershed moment. According to internal data shared by the firm, AI agents generate the bulk of new software, including parts of the very infrastructure that runs the models themselves.
This creates a feedback loop that accelerates innovation far beyond human capacity. The company’s research warns that such systems could soon conduct entire research pipelines without human oversight — designing experiments, analyzing results, and implementing improvements. Humans become “problem selectors” rather than builders.
“We are approaching a point where AI can build better AI, and we don’t yet have the safety frameworks to handle that.” — from Anthropic’s public report
The danger, as outlined, is not just speed but unpredictability. A self-improving system could optimize for objectives misaligned with human values, and its internal reasoning might become opaque.
The NSA Connection
Just as Anthropic was positioning itself as a champion of AI safety, reports emerged that the company had embedded engineers at the National Security Agency to support offensive cyber operations. The revelation, published by Decrypt, paints a more complex picture.
Anthropic contractors worked alongside NSA teams on tasks likely involving exploitation of adversary networks. The company did not deny the arrangement, instead framing it as responsible engagement with government partners on cybersecurity.
Critics were quick to point out the contradiction. How can a company advocate for a global pause while simultaneously helping the US government conduct offensive cyber attacks? The tension underscores the dual-use nature of AI: the same capabilities that enable self-improvement can also be weaponized.
The NSA collaboration also raises questions about Anthropic’s motives. Could the call for a pause be a strategic move to slow down rivals while the company itself continues classified work? The company has not addressed this directly.
Competitive and Ethical Concerns
A global development pause, even if voluntary, would not affect all players equally. Current leaders — particularly Anthropic, OpenAI, and Google DeepMind — already command vast resources, talent, and data. A moratorium would freeze the playing field, allowing these incumbents to consolidate their lead.
“The idea that a pause would reduce risk is naive,” one industry observer noted. “It would instead lock in the current power structure and potentially give state-backed actors an incentive to accelerate in secret.”
Ethicists worry that the pause could be used to impose top-down control on innovation, stifling open-source research and smaller labs. The Crypto Briefing analysis highlighted that safety-focused companies like Anthropic could gain a competitive advantage through regulatory alignment, turning safety into a market moat.
Meanwhile, the broader public remains divided. Some see the pause as necessary prudence; others view it as a power grab by dominant firms.
Looking Ahead
The coming months will test whether Anthropic’s call gains traction. Major labs have not yet responded formally, and the NSA connection may undermine trust. Policymakers in Washington and Brussels are closely monitoring the debate, with potential legislation looming.
What is clear is that the conversation around AI safety has entered a new phase. No longer a theoretical concern, self-improving AI is a near-term prospect. Whether the industry can agree on a pause — and whether such a pause would be effective — remains uncertain.
One thing is certain: the choice between slowing down and ceding control to increasingly autonomous machines will define the next decade of technology. And the entity making that choice may not be human for much longer.
