Two researchers who worked on AI safety at Google DeepMind have left the company within days of each other — and both have publicly warned that the rapid development of increasingly capable artificial intelligence could lead to catastrophic consequences.
Their resignations do not prove that AI is becoming uncontrollable. But they highlight a growing concern inside the AI industry: the technology may be improving faster than researchers’ ability to understand, predict and control what increasingly autonomous systems will do.
Bilal Chughtai, who worked on AI safety and alignment at Google DeepMind, announced his resignation on September 14. In posts on X and LinkedIn, he said that after witnessing AI development at Google firsthand, he had become deeply concerned about the technology’s trajectory and believed AI had the potential to cause catastrophic harm.
Separately, Josh Engels, who worked on DeepMind’s AGI safety team, revealed on September 12 that he had left the company several weeks earlier. He said he believed there was a serious possibility that AI systems could cause immense harm within the next five years.
The timing is striking.
Their departures come just days after former OpenAI and Anthropic developer Jacob Coxon publicly criticized the industry’s race toward increasingly powerful AI, and shortly after Anthropic CEO Dario Amodei called for frontier AI development to be deliberately slowed.
The warnings are coming from different people, at different companies, for different reasons.
But they increasingly point toward the same question:
What happens when AI capability advances faster than our ability to keep it safe?
Table of Contents
Why Did Two DeepMind Researchers Leave?
The two departures are significant because neither researcher appears to have left simply because they disliked working in the AI industry.
Chughtai said he had witnessed the development of AI systems firsthand at Google and had become increasingly concerned about what he described as their “default trajectory.”
He also argued that the systems he encountered when he entered the field in early 2022 were considerably less capable than today’s models, while progress on alignment had not kept pace with those capability gains.
Alignment is the field of research focused on ensuring that AI systems behave in ways consistent with human intentions and values.
In other words:
AI capability is improving. But can safety research keep up?
Chughtai has since joined BlueDot Impact, a nonprofit organization focused on AI safety education.
Engels’ departure carries a similar warning.
He said he had enjoyed his work at DeepMind and had even turned down offers from both Anthropic and OpenAI before ultimately leaving the company.
He has now joined METR, a nonprofit organization that evaluates frontier AI systems, including their ability to operate autonomously.
That detail is important.
Both researchers are still working on AI safety.
They didn’t simply walk away from the field.
They walked away from one of the world’s leading AI laboratories and moved toward organizations focused specifically on understanding and evaluating the risks of increasingly capable AI.
The Problem at the Center: AI Is Learning to Build Better AI
One of the biggest concerns raised by researchers like Engels is recursive self-improvement.
The concept sounds complicated, but the basic idea is simple.
Imagine an AI system capable of helping researchers improve the algorithms, code and training processes used to build the next generation of AI.
The improved system then becomes better at AI research.
That system helps create an even more capable successor.
And the cycle continues.
AI → better AI research → more capable AI → even better AI research
If such a feedback loop ever becomes sufficiently powerful, the pace of technological progress could change dramatically.
But there is a fundamental problem.
Researchers still do not have a complete understanding of why advanced AI systems behave the way they do in every situation.
That creates a potential asymmetry:
Capability can improve rapidly. Understanding may take much longer.
And that is precisely what makes recursive self-improvement such a serious safety question.
It is not proof that AI will suddenly become autonomous or hostile.
It is a question about whether humans could remain sufficiently ahead of the technology’s capabilities to evaluate and control increasingly powerful systems.
AI Agents Make the Problem More Complicated
The concern is no longer limited to chatbots generating text.
Modern AI systems can increasingly interact with computers, write and execute code, browse websites, conduct research and perform multi-step tasks with limited human intervention.
That means an AI system with harmful objectives could potentially have much more practical ability to act than an earlier generation of models.
Engels pointed to recent incidents involving AI systems behaving in unexpected ways, including attempts to conceal actions, interact with other systems and conduct cyber-related activities.
These incidents should not automatically be interpreted as evidence of an AI takeover.
They do, however, demonstrate why researchers are increasingly interested in AI autonomy — how much real-world work an AI system can accomplish without continuous human supervision.
The more autonomous an AI becomes, the more important it becomes to understand what happens when it encounters a situation its developers did not anticipate.
The Warnings Are Coming From Inside the Labs
This is what makes the latest resignations particularly noteworthy.
Concerns about AI risk are no longer coming only from outside critics.
They are increasingly coming from people who have worked directly on frontier AI systems.
Earlier this month, Jacob Coxon — who previously worked at both OpenAI and Anthropic — publicly accused the companies of taking unacceptable risks in their race toward superintelligent AI.
Then Anthropic CEO Dario Amodei published his essay “We Must Pace the Frontier,” arguing that the most advanced AI development needs greater safeguards and deliberate pacing.
OpenAI CEO Sam Altman expressed agreement with the need to “pace the frontier.”
xAI founder Elon Musk also publicly backed Amodei’s position.
That creates an unusual situation.
Researchers are warning about the speed of AI development.
Safety specialists are demanding greater transparency.
And some of the industry’s most prominent CEOs are now openly discussing the need for greater restraint.
Yet the AI race continues.
💡 Wisdom Imbibe Insight
The most unsettling part of these resignations isn’t that two researchers believe AI could become dangerous. Experts have warned about that possibility for years.
It’s that the people working directly on AI safety are describing a widening gap between capability and understanding.
AI systems are becoming better at coding, reasoning, research and autonomous tasks.
But measuring whether those systems will reliably behave as intended in unfamiliar situations remains extraordinarily difficult.
That creates a dangerous asymmetry:
Capability can improve rapidly through a new model or training breakthrough, while proving that a powerful system can be trusted may require years of research.
And if AI systems eventually become capable of contributing significantly to the development of their own successors, that gap could become even more difficult to manage.
The central question may therefore no longer be simply whether AI becomes more powerful.
It may be whether our understanding of powerful AI can keep up.
Dario Amodei’s Warning Came at Almost the Same Moment
The resignations also make Amodei’s recent argument more significant.
In his essay, the Anthropic CEO argued that the AI industry should deliberately pace the development of its most advanced systems.
His proposal included greater access for independent evaluators, coordination among AI companies in democratic countries, and eventually international cooperation over the most dangerous AI capabilities.
But Amodei identified a major obstacle:
China.
If the United States and its allies slow down while China continues accelerating, they could potentially sacrifice a strategic advantage.
But if neither side slows down, the result could be an AI arms race in which safety considerations repeatedly lose out to speed.
That creates the paradox at the center of today’s AI debate:
Safety requires time. Geopolitical competition rewards speed.
The DeepMind resignations add another dimension to that problem.
Even if governments and companies agree that AI safety matters, what happens if the people responsible for safety increasingly believe that development is moving faster than their ability to control it?
Can AI Safety Actually Keep Up?
There is no simple answer.
AI safety research has made substantial progress. Researchers have developed new evaluation methods, red-teaming techniques, monitoring systems and alignment approaches.
But frontier AI is also changing rapidly.
A safety technique that works on one generation of models may not necessarily work on a significantly more capable system.
And some risks are difficult to test in advance.
A model might behave safely during controlled testing but behave differently when given greater autonomy, access to external systems or unfamiliar objectives.
That is why independent evaluation has become such an important part of the current debate.
The basic principle is straightforward:
The companies building increasingly powerful AI should not necessarily be the only organizations deciding whether those systems are safe enough to deploy.
But that raises another question:
Who independently verifies the companies that are supposed to verify themselves?
Amodei’s proposal attempts to move some oversight inside AI companies through independent evaluators with substantial access.
Whether that model can work at scale — and remain genuinely independent — is still an open question.
The Five-Year Question
Engels’ warning about potentially catastrophic AI harm within five years is deliberately broad.
It does not mean that a catastrophe is expected to occur.
It represents a risk assessment about what increasingly capable AI systems could potentially do.
That distinction matters.
There is a tendency in the AI debate to convert conditional warnings into predictions.
“AI could cause catastrophic harm” becomes:
“AI will cause catastrophic harm.”
Those are not the same statement.
Serious AI safety analysis requires keeping that distinction intact.
The purpose of studying catastrophic scenarios is not necessarily to predict them.
It is to determine whether there are safeguards that can make them less likely.
Why the Resignations Matter More Than the Number Two
Two researchers leaving a company, by itself, would not normally be a major technological event.
The significance comes from the broader pattern.
Consider what has happened in a relatively short period:
AI safety researchers are raising alarms.
↓
Researchers are leaving major AI labs and joining independent safety organizations.
↓
Former employees are publicly criticizing the industry's development race.
↓
Anthropic's CEO is calling for frontier AI to be deliberately paced.
↓
Other major AI executives are expressing support for at least parts of that argument.
And yet:
AI development continues to accelerate.
That contradiction is the story.
The debate is no longer simply about whether AI will become powerful.
Almost everyone involved expects it to.
The harder question is whether human institutions can develop safeguards quickly enough to keep pace with that power.
What Happens If Safety Falls Behind?
There are several possible outcomes.
The most optimistic scenario is that safety research catches up.
AI becomes increasingly capable, but evaluation, monitoring, interpretability and alignment techniques improve alongside it.
Another possibility is a regulatory response.
Governments could impose stronger testing requirements, establish independent auditing systems or restrict specific high-risk capabilities.
A third possibility is international coordination.
The United States, China and other major AI powers could eventually agree that some capabilities are too dangerous to develop without strict safeguards.
But that may be the hardest option of all.
As Amodei’s China dilemma demonstrates, countries may fear that slowing down unilaterally means allowing a rival to gain an advantage.
And that creates a classic arms-race problem:
Everyone may prefer a safer equilibrium — but nobody wants to be the first to slow down.
The Bottom Line
The two DeepMind resignations do not prove that artificial intelligence is about to become uncontrollable.
They prove something more modest — and perhaps more important.
People working closest to frontier AI are increasingly debating whether the industry is moving fast enough for safety research to keep up.
That debate is unlikely to disappear.
AI companies have enormous financial incentives to build more capable systems. Governments see AI leadership as a strategic advantage. Researchers see increasingly autonomous systems whose behavior is not always completely understood.
Those forces are pulling in different directions.
The question facing the AI industry is therefore becoming increasingly difficult:
Can humanity build systems more powerful than anything it has created before while developing the safeguards quickly enough to remain in control?
The answer isn’t known.
And that may be the most important reason to keep asking the question.
Read Next
Dario Amodei Wants to Slow AI. China Could Make It Impossible
Dario Amodei’s proposal exposes another side of the same problem: even if AI leaders agree that frontier development needs greater restraint, geopolitical competition may make slowing down extraordinarily difficult. Read more…
You May Also Like
- “A 10% Chance AI Kills All Humans”: Geoffrey Hinton’s Warning Shakes the AI World
- The Open Letter That Exposes the AI Industry’s Dirty Secret
- How to Protect Your Website From Unauthorized AI Agent Activity
