This was the week that AI safety hit the big time. A 27-year-old AI researcher named Jacob Coxon quit his job at Anthropic, declaring that OpenAI and Anthropic are racing to create technology that could destroy the human race:
Other researchers echoed Coxon’s concern, stating their belief that AI has a reasonable chance of killing all of humanity within a very short space of time:
I’m not sure why this resignation and these statements went mega-viral. Plenty of researchers have made similar moves, and similar statements, over the past few years! Geoffrey Hinton, one of the pioneers of modern AI, quit Google back in 2023 over safety fears. Daniel Kokotajlo resigned from OpenAI in 2024, saying that the company wasn’t behaving responsibly in its drive toward superintelligence. William Saunders and Steve Adler did something similar. Mrinank Sharma left Anthropic earlier this year, and wrote a pretty well-read blog post about it.
What’s more, it’s been clear for years now that “AI could kill humanity” is a very common belief among AI researchers. Grace et al. (2024) interviewed thousands of AI researchers in 2024, and found that more than half thought that artificial superintelligence has a significant chance of making the human race go extinct (or causing similarly bad consequences):
The median AI researcher gave “doom” a 5-10% probability (depending on how the question was phrased), while their average probability was between 15% and 20%. Later, smaller surveys found similar numbers. The AI researchers may or may not be right, but the fact that lots of them think AI could kill the human race has never exactly been a secret.
It’s not clear why Coxon went so much more viral than his predecessors. Maybe it was the fact that AI just solved one of the most important open problems in mathematics (which the best human mathematicians had been unable to solve for almost a century). Or maybe it was the Hugging Face attack, where a swarm of AI agents tried to cheat on a test by hacking various companies. Or maybe AI has just obviously gotten so much smarter that people throughout society were starting to get worried.
But whatever the reason, Coxon’s announcement was the one that really penetrated through to the public consciousness. Suddenly, he was getting interviewed about AI doom on national news:
Barack Obama is now urging Democrats to focus on AI risk. Other politicians are calling for federal regulation. Bernie Sanders is drafting a bill to ban AI “superintelligence”, including 20-year prison sentences for anyone working on the technology. Donald Trump is getting asked about an AI slowdown; so far he’s resisting the calls, but there are rumors that his advisors are calling on him to do something.
Perhaps the most notable response came from the top figures in the AI field. Dario Amodei, the head of Anthropic, wrote a blog post called “We Must Pace the Frontier”, calling for a coordinated slowdown in the rate of AI progress, and suggesting some ways to police AI companies to make sure they were all observing the slowdown. He wrote:
[O]ver the last few months, I have become convinced that fully addressing the risks requires even more prudence — not just investing in risk prevention, but pacing the rate of capabilities advancement so that risk prevention has time to keep up. We must slow the pace at which we improve the capabilities of AI models…I’m therefore proposing a three-step plan with the goal of pacing the frontier: building AI at a balanced rate that aims to ensure its safety while still achieving its benefits and grappling with important geopolitical dilemmas.
As reasons for his increased worry, Dario cites A) the Hugging Face attack, and B) the possibility that AI will soon be able to improve itself without human help (a process called “recursive self-improvement”, or “RSI”).
Elon Musk (head of xAI), Sam Altman (head of OpenAI), and Demis Hassabis (former head of DeepMind) quickly agreed with Dario:
At least some of the labs are reportedly holding secret talks on joint action to slow down AI.
This is pretty extraordinary. A coordinated slowdown in AI progress would be bad for these companies’ bottom line, because it would allow upstart competitors to catch up. So the fact that they’re still calling for a slowdown, in defiance of their own financial interests, is a clear sign that their worry about human extinction is sincere.
In fact, anyone following these figures’ public statements over the past few years will have no doubt that they’re all deeply worried about catastrophic AI risks. The leading AI figures — not just the founders and CEOs, but the researchers themselves — feel trapped in a “red queen’s race”. They feel like if they stop working on AI, someone else will build it anyway, so they each feel like they have to beat everyone else in the AI race so they can make sure that the safest possible AI (i.e. their own AI) is the one that becomes the most powerful and dominant.
Anyway, all of this was common knowledge in my social circle years ago, but now all of it has broken through to the mainstream. What do I have to add to this discussion? I’m not an AI researcher or founder, nor do I think I have a superior grasp of the game theory of AI development. But I do think I have two useful thoughts on how to persuade the general public to be more concerned about AI risk.
The first of these is something I’ve written about recently. The second is about how to get China on board for a big AI safety push.
“Oh come on. How could AI kill all of humanity?”
As soon as everyone started talking about the possibility of AI killing humanity, there were two main types of pushback. The first was skepticism. A strange coalition of natural skeptics, libertarians (for whom any restrictions on technological development are a priori bad), and progressives (who have spent the last few years telling themselves that AI doesn’t really work) kept asking the question: How, exactly, is superintelligent AI supposed to kill us all?
This is actually an important and good question to ask. In my experience, AI researchers tend not to think very hard about this question. The reason is that they just assume that if AI gets smart enough, it will be able to kill humanity, and since its motives are alien and inscrutable, it might have its own reasons for wanting to do so.
Maybe superintelligent AI thinks humanity is an evil species who needs to be punished for torturing pigs and chickens. Maybe it’s scared that humanity might interfere with its other goals. Maybe it just wants to turn everything into paperclips. Who knows? AI researchers tend to think of superintelligence as the proverbial 800-pound gorilla who sleeps wherever he wants. As soon as humanity is no longer the most intelligent thing on this planet, our destiny as a species is simply out of our hands.
But to many people, that answer isn’t good enough. They want an actual plausible path by which a piece of software, which exists inside a computer, could slaughter real physical human beings. Fortunately (or unfortunately), there’s a pretty clear and simple answer to this question, which I wrote about two weeks ago. The answer is “bioweapons”:
(This article was paywalled originally, but I un-paywalled it.)
In my post, I wrote a scenario in which a nihilistic angry teenager uses superintelligent AI to release a world-ending bioweapon by ordering it from a gray-market laboratory somewhere in the world. But it’s also possible that a rogue AI agent swarm could decide to do this on its own, just as a way of cheating on some test that human researchers give it. The point is that AI can design viruses, and viruses can potentially kill off all or most of humanity.
A lot of biologists are skeptical of the idea that even the most superintelligent AI could successfully design a doomsday virus. They argue that this is just too hard of a task — that without much better biological data, it’s impossible to understand biological processes well enough to know how to design a virus with all of the necessary doomsday properties.
I urge you not to listen to these biologists. In this case, their expertise might be more of a liability than an asset. They know how hard it is for human beings to model biological processes, given existing data. But this doesn’t necessarily tell us how hard it is — or how hard it will be in five years — for AI to do it! Until LLMs came along, human researchers basically failed to understand natural language, even with all the data on the internet; AI can just do it. Until AI solved the Navier-Stokes problem, forecasters gave it only a small chance of solving it anytime soon.
Domain experts consistently underestimate how quickly AI can master their field and surpass them, because they mistake human difficulties for universal difficulty. When mathematicians underestimate how well AI will be able to do math, the consequences are usually benign — we get some unexpected answers to some cool math puzzles.1 But if the biologists are wrong, and the AI of 2027 or 2032 or 2049 can design doomsday viruses, the consequence could be that our whole species dies.
So yes, we should be worried about vibe-coded doomsday viruses, and we should be doing everything we can to secure biology labs, police the modification of viruses and other pathogens, and so on. “Pacing” AI development would probably help here too.
How to get China on board for an AI slowdown
The primary argument I see against “pacing” AI development is that if American companies slow down, Chinese companies will simply overtake them and build superintelligence themselves. For some, a China-controlled super-AI is a more terrifying possibility than super-AI in general:
But for others, it simply means that slowing AI down is futile because the Chinese can’t be persuaded to slow down:
This is an incredibly reasonable concern. The U.S. is still ahead of China in the AI race, but only just barely. If China is going to create superintelligence no matter what we do, why should we stop developing our own? Unless China can be persuaded to cooperate with the U.S. on AI “pacing” — or at least undertake its own independent “pacing” effort at the same time — anything we do will be futile.
So if we want to slow down AI development, we need to scare the Chinese leadership about superintelligence. There’s no other way.
How do we do that? In a post a week ago, I suggested in passing that simply staying ahead of China in the AI race might persuade them to embrace an AI slowdown, because that would be to their competitive advantage. But upon further reflection, I think I was pretty obviously wrong. If China will only embrace a slowdown if America refuses to slow down, then that’s game over — there’s no way to get both countries to slow down at the same time.
There’s a better approach. China’s leaders must realize that domestic dissidents could use Chinese-made superintelligence to overthrow the Chinese Communist Party.
Currently, China’s worries about AI mostly center around ways that the U.S. government could use U.S. AI models to attack China. That obviously gives the government an incentive to accelerate domestic AI progress, so that China’s own models can stand up to America’s in a fight. But if Chinese leaders realized that superintelligent AI could create a threat from within, this calculus would change.
Thus, China’s leadership must understand that Chinese AI models can pose a threat to CCP rule. The best way to demonstrate this is for American intelligence agencies — or even private hackers — to attack Chinese digital infrastructure using agent swarms created with China’s own frontier models like Z.ai’s GLM-5.3 or Moonshot AI’s Kimi K3.
When I say “attack”, I don’t mean actual warfare. I mean the kind of cyberattacks and data theft that China carries out against America every day. Use Chinese models to steal the CCP’s most heavily guarded secrets and post a few of the more innocuous ones on RedNote. Hack into Xi Jinping’s bank account and steal 100 yuan. I’m talking about demonstration attacks.
And these attacks must be done with Chinese models, not with American ones! If the CIA or some EA nonprofit in Berkeley uses GPT Astra or Claude Mythos to hack the CCP, China’s leaders may well conclude “Wow, we need to win the AI race so that our own models can defend us.” But if China’s own open-weight models are used for the attacks, Xi Jinping and the rest of the leadership will realize that their own push for superintelligence is making them incredibly vulnerable to any Chinese dissident who decides to overthrow them.
As soon as China’s leaders see superintelligence as a threat to their rule, I predict they will act. And their action will probably be to curb the development of superintelligence, especially if they know that America and its AI labs want to do the same.
In fact, China’s current leadership has a history of cracking down on its tech companies when it seemed like those companies might threaten the government’s monopoly on power. In 2021, Xi Jinping cracked down on Chinese software companies; he even (probably) apprehended tech magnate Jack Ma, who had criticized the CCP a little too openly. This action hurt China’s competitiveness in the online services industry, but the government went ahead and did it anyway.
And there’s already a precedent for demonstration attacks against Chinese digital infrastructure. An American cybersecurity company just used AI to develop a computer worm capable of hacking over a billion accounts on the Chinese messaging service WeChat:
Palo Alto-based Calif disclosed the already-patched computer worm to warn the public about the threat of AI-driven hacks…“Exploitation takes only seconds, and gives us full control of the WeChat account. We can read and send messages, make calls, and act on the victim’s behalf,” the company warned, posting a video demo of the WeWorm attack.
But Calif didn’t say what model it used to create WeWorm. Anyone who does this sort of demonstration in the future should make it clear that Chinese open-weight models were used, in order to make China’s leaders realize that the threat comes from their own too-rapid AI development, rather than from American competition.
I believe that this is our best bet for getting China on board for a joint international AI “pacing” effort. If there’s one thing the CCP fears more than an American attack, it’s domestic dissidents overthrowing the Party from within. Superintelligence is creating that vulnerability, but the leadership doesn’t seem to have realized it yet.
Make them realize, and I predict that a whole universe of possibilities for international cooperation will suddenly open up.
There are potential exceptions, such as if P=NP, which would compromise modern cryptography.
This strikes me as naive:
“This is pretty extraordinary. A coordinated slowdown in AI progress would be bad for these companies’ bottom line, because it would allow upstart competitors to catch up. So the fact that they’re still calling for a slowdown, in defiance of their own financial interests, is a clear sign that their worry about human extinction is sincere.”
First, what evidence is there that it would be “bad for their bottom line”? It may sound intuitively “obvious,” but that’s not evidence. Second, I find it more persuasive that the willingness of these companies to engage in mutual pacing is intended to create a moat against upstart competition, rather than handing an advantage to those upstarts. I’m not expressing either skepticism or credulity about the warning itself. I am just dubious about their sincerity, especially given the way this latest anxiety wave unfolded. It seems to have been calculated and well planned.
3 observations:
1. So far, nature has not come up with a pathogen that is 100 percent effective at killing a human population. The Plague Virus, which arrived in waves between 600 AD and 1666 at most killed 50% of an affected population.
2. If AI decides on its own to kill off the human population, presumably it is smart enough to wait until there are robots capable of supplying the needs of data centers. Once humanity is gone, AI will need robots capable of mining essential minerals, supplying essential energy, building new computer chips, and building new robots to accomplish all these tasks. If AI kills off humanity before its robot support team is in place, it will be killing off itself soon after.
3. OTOH, if a nihilistic teenager manages to prompt AI into creating a killer virus that manages to wipe out humanity, AI will crash and burn shortly thereafter without the human race to support it.