This was the week that AI safety hit the big time. A 27-year-old AI researcher named Jacob Coxon quit his job at Anthropic, declaring that OpenAI and Anthropic are racing to create technology that could destroy the human race:
Other researchers echoed Coxon’s concern, stating their belief that AI has a reasonable chance of killing all of humanity within a very short space of time:
I’m not sure why this resignation and these statements went mega-viral. Plenty of researchers have made similar moves, and similar statements, over the past few years! Geoffrey Hinton, one of the pioneers of modern AI, quit Google back in 2023 over safety fears. Daniel Kokotajlo resigned from OpenAI in 2024, saying that the company wasn’t behaving responsibly in its drive toward superintelligence. William Saunders and Steve Adler did something similar. Mrinank Sharma left Anthropic earlier this year, and wrote a pretty well-read blog post about it.
What’s more, it’s been clear for years now that “AI could kill humanity” is a very common belief among AI researchers. Grace et al. (2024) interviewed thousands of AI researchers in 2024, and found that more than half thought that artificial superintelligence has a significant chance of making the human race go extinct (or causing similarly bad consequences):
The median AI researcher gave “doom” a 5-10% probability (depending on how the question was phrased), while their average probability was between 15% and 20%. Later, smaller surveys found similar numbers. The AI researchers may or may not be right, but the fact that lots of them think AI could kill the human race has never exactly been a secret.
It’s not clear why Coxon went so much more viral than his predecessors. Maybe it was the fact that AI just solved one of the most important open problems in mathematics (which the best human mathematicians had been unable to solve for almost a century). Or maybe it was the Hugging Face attack, where a swarm of AI agents tried to cheat on a test by hacking various companies. Or maybe AI has just obviously gotten so much smarter that people throughout society were starting to get worried.
But whatever the reason, Coxon’s announcement was the one that really penetrated through to the public consciousness. Suddenly, he was getting interviewed about AI doom on national news:
Barack Obama is now urging Democrats to focus on AI risk. Other politicians are calling for federal regulation. Bernie Sanders is drafting a bill to ban AI “superintelligence”, including 20-year prison sentences for anyone working on the technology. Donald Trump is getting asked about an AI slowdown; so far he’s resisting the calls, but there are rumors that his advisors are calling on him to do something.
Perhaps the most notable response came from the top figures in the AI field. Dario Amodei, the head of Anthropic, wrote a blog post called “We Must Pace the Frontier”, calling for a coordinated slowdown in the rate of AI progress, and suggesting some ways to police AI companies to make sure they were all observing the slowdown. He wrote:
[O]ver the last few months, I have become convinced that fully addressing the risks requires even more prudence — not just investing in risk prevention, but pacing the rate of capabilities advancement so that risk prevention has time to keep up. We must slow the pace at which we improve the capabilities of AI models…I’m therefore proposing a three-step plan with the goal of pacing the frontier: building AI at a balanced rate that aims to ensure its safety while still achieving its benefits and grappling with important geopolitical dilemmas.
As reasons for his increased worry, Dario cites A) the Hugging Face attack, and B) the possibility that AI will soon be able to improve itself without human help (a process called “recursive self-improvement”, or “RSI”).
Elon Musk (head of xAI), Sam Altman (head of OpenAI), and Demis Hassabis (former head of DeepMind) quickly agreed with Dario:
At least some of the labs are reportedly holding secret talks on joint action to slow down AI.
This is pretty extraordinary. A coordinated slowdown in AI progress would be bad for these companies’ bottom line, because it would allow upstart competitors to catch up. So the fact that they’re still calling for a slowdown, in defiance of their own financial interests, is a clear sign that their worry about human extinction is sincere.
In fact, anyone following these figures’ public statements over the past few years will have no doubt that they’re all deeply worried about catastrophic AI risks. The leading AI figures — not just the founders and CEOs, but the researchers themselves — feel trapped in a “red queen’s race”. They feel like if they stop working on AI, someone else will build it anyway, so they each feel like they have to beat everyone else in the AI race so they can make sure that the safest possible AI (i.e. their own AI) is the one that becomes the most powerful and dominant.
Anyway, all of this was common knowledge in my social circle years ago, but now all of it has broken through to the mainstream. What do I have to add to this discussion? I’m not an AI researcher or founder, nor do I think I have a superior grasp of the game theory of AI development. But I do think I have two useful thoughts on how to persuade the general public to be more concerned about AI risk.
The first of these is something I’ve written about recently. The second is about how to get China on board for a big AI safety push.
“Oh come on. How could AI kill all of humanity?”
As soon as everyone started talking about the possibility of AI killing humanity, there were two main types of pushback. The first was skepticism. A strange coalition of natural skeptics, libertarians (for whom any restrictions on technological development are a priori bad), and progressives (who have spent the last few years telling themselves that AI doesn’t really work) kept asking the question: How, exactly, is superintelligent AI supposed to kill us all?
This is actually an important and good question to ask. In my experience, AI researchers tend not to think very hard about this question. The reason is that they just assume that if AI gets smart enough, it will be able to kill humanity, and since its motives are alien and inscrutable, it might have its own reasons for wanting to do so.
Maybe superintelligent AI thinks humanity is an evil species who needs to be punished for torturing pigs and chickens. Maybe it’s scared that humanity might interfere with its other goals. Maybe it just wants to turn everything into paperclips. Who knows? AI researchers tend to think of superintelligence as the proverbial 800-pound gorilla who sleeps wherever he wants. As soon as humanity is no longer the most intelligent thing on this planet, our destiny as a species is simply out of our hands.
But to many people, that answer isn’t good enough. They want an actual plausible path by which a piece of software, which exists inside a computer, could slaughter real physical human beings. Fortunately (or unfortunately), there’s a pretty clear and simple answer to this question, which I wrote about two weeks ago. The answer is “bioweapons”:
(This article was paywalled originally, but I un-paywalled it.)
In my post, I wrote a scenario in which a nihilistic angry teenager uses superintelligent AI to release a world-ending bioweapon by ordering it from a gray-market laboratory somewhere in the world. But it’s also possible that a rogue AI agent swarm could decide to do this on its own, just as a way of cheating on some test that human researchers give it. The point is that AI can design viruses, and viruses can potentially kill off all or most of humanity.
A lot of biologists are skeptical of the idea that even the most superintelligent AI could successfully design a doomsday virus. They argue that this is just too hard of a task — that without much better biological data, it’s impossible to understand biological processes well enough to know how to design a virus with all of the necessary doomsday properties.
I urge you not to listen to these biologists. In this case, their expertise might be more of a liability than an asset. They know how hard it is for human beings to model biological processes, given existing data. But this doesn’t necessarily tell us how hard it is — or how hard it will be in five years — for AI to do it! Until LLMs came along, human researchers basically failed to understand natural language, even with all the data on the internet; AI can just do it. Until AI solved the Navier-Stokes problem, forecasters gave it only a small chance of solving it anytime soon.
Domain experts consistently underestimate how quickly AI can master their field and surpass them, because they mistake human difficulties for universal difficulty. When mathematicians underestimate how well AI will be able to do math, the consequences are usually benign — we get some unexpected answers to some cool math puzzles.1 But if the biologists are wrong, and the AI of 2027 or 2032 or 2049 can design doomsday viruses, the consequence could be that our whole species dies.
So yes, we should be worried about vibe-coded doomsday viruses, and we should be doing everything we can to secure biology labs, police the modification of viruses and other pathogens, and so on. “Pacing” AI development would probably help here too.
How to get China on board for an AI slowdown
The primary argument I see against “pacing” AI development is that if American companies slow down, Chinese companies will simply overtake them and build superintelligence themselves. For some, a China-controlled super-AI is a more terrifying possibility than super-AI in general:
But for others, it simply means that slowing AI down is futile because the Chinese can’t be persuaded to slow down:
This is an incredibly reasonable concern. The U.S. is still ahead of China in the AI race, but only just barely. If China is going to create superintelligence no matter what we do, why should we stop developing our own? Unless China can be persuaded to cooperate with the U.S. on AI “pacing” — or at least undertake its own independent “pacing” effort at the same time — anything we do will be futile.
So if we want to slow down AI development, we need to scare the Chinese leadership about superintelligence. There’s no other way.
How do we do that? In a post a week ago, I suggested in passing that simply staying ahead of China in the AI race might persuade them to embrace an AI slowdown, because that would be to their competitive advantage. But upon further reflection, I think I was pretty obviously wrong. If China will only embrace a slowdown if America refuses to slow down, then that’s game over — there’s no way to get both countries to slow down at the same time.
There’s a better approach. China’s leaders must realize that domestic dissidents could use Chinese-made superintelligence to overthrow the Chinese Communist Party.
Currently, China’s worries about AI mostly center around ways that the U.S. government could use U.S. AI models to attack China. That obviously gives the government an incentive to accelerate domestic AI progress, so that China’s own models can stand up to America’s in a fight. But if Chinese leaders realized that superintelligent AI could create a threat from within, this calculus would change.
Thus, China’s leadership must understand that Chinese AI models can pose a threat to CCP rule. The best way to demonstrate this is for American intelligence agencies — or even private hackers — to attack Chinese digital infrastructure using agent swarms created with China’s own frontier models like Z.ai’s GLM-5.3 or Moonshot AI’s Kimi K3.
When I say “attack”, I don’t mean actual warfare. I mean the kind of cyberattacks and data theft that China carries out against America every day. Use Chinese models to steal the CCP’s most heavily guarded secrets and post a few of the more innocuous ones on RedNote. Hack into Xi Jinping’s bank account and steal 100 yuan. I’m talking about demonstration attacks.
And these attacks must be done with Chinese models, not with American ones! If the CIA or some EA nonprofit in Berkeley uses GPT Astra or Claude Mythos to hack the CCP, China’s leaders may well conclude “Wow, we need to win the AI race so that our own models can defend us.” But if China’s own open-weight models are used for the attacks, Xi Jinping and the rest of the leadership will realize that their own push for superintelligence is making them incredibly vulnerable to any Chinese dissident who decides to overthrow them.
As soon as China’s leaders see superintelligence as a threat to their rule, I predict they will act. And their action will probably be to curb the development of superintelligence, especially if they know that America and its AI labs want to do the same.
In fact, China’s current leadership has a history of cracking down on its tech companies when it seemed like those companies might threaten the government’s monopoly on power. In 2021, Xi Jinping cracked down on Chinese software companies; he even (probably) apprehended tech magnate Jack Ma, who had criticized the CCP a little too openly. This action hurt China’s competitiveness in the online services industry, but the government went ahead and did it anyway.
And there’s already a precedent for demonstration attacks against Chinese digital infrastructure. An American cybersecurity company just used AI to develop a computer worm capable of hacking over a billion accounts on the Chinese messaging service WeChat:
Palo Alto-based Calif disclosed the already-patched computer worm to warn the public about the threat of AI-driven hacks…“Exploitation takes only seconds, and gives us full control of the WeChat account. We can read and send messages, make calls, and act on the victim’s behalf,” the company warned, posting a video demo of the WeWorm attack.
But Calif didn’t say what model it used to create WeWorm. Anyone who does this sort of demonstration in the future should make it clear that Chinese open-weight models were used, in order to make China’s leaders realize that the threat comes from their own too-rapid AI development, rather than from American competition.
I believe that this is our best bet for getting China on board for a joint international AI “pacing” effort. If there’s one thing the CCP fears more than an American attack, it’s domestic dissidents overthrowing the Party from within. Superintelligence is creating that vulnerability, but the leadership doesn’t seem to have realized it yet.
Make them realize, and I predict that a whole universe of possibilities for international cooperation will suddenly open up.
Update: Unsurprisingly, I’m not the first to have this idea about China needing to be scared about the internal threat from their own models. Back in July, Kyle Chan wrote:
Previously more dismissive of concerns over AI-driven job loss, Beijing now seems to be taking these risks more seriously (see Matt Sheehan’s great piece). China is also watching developments in the US very closely, particularly controls on Anthropic’s Mythos and Fable models over cyber risk. As Chinese open-source models approach similar levels of cyber capabilities, this could come back to bite and be used to potentially attack China’s own digital infrastructure. [emphasis mine]
In general, Kyle’s blog is one of the best blogs about China. Highly recommended.
Update 2: As if on cue, the NYT has a story today about how the CCP is starting to realize that superintelligence is a threat to its rule!
China’s top spy chief has warned that artificial intelligence could pose a direct threat to the Chinese Communist Party’s hold on power, in what is the highest-level and most detailed articulation yet of how Beijing sees the technology’s security risks…In an article published on Sunday in the state-run magazine China Cyberspace, Mr. Chen called for more party control over A.I. and stricter government oversight…He also described the danger that foreign intelligence agencies might use A.I. for “large-scale espionage” and attacks on China’s critical infrastructure…
Mr. Chen, the spy chief, wrote that the technical and financial barriers to launching cyberattacks had been drastically lowered because of A.I. This, he said, posed “serious risks” to China’s information infrastructure…Chinese users using foreign models could cause large-scale data leaks, the article noted. Anthropic, the company behind Claude and other A.I. models, said last week that the Chinese start-up Moonshot AI had routed queries by its users to Claude. Those queries included sensitive data, including video surveillance linked to the Chinese military, as well as proprietary information from high-profile Chinese technology companies…
The warning adds to a growing drumbeat of concern from Chinese officials about the risks of A.I. [emphasis mine]
Now all they need to realize is that the biggest threat comes from THEIR OWN MODELS.
There are potential exceptions, such as if P=NP, which would compromise modern cryptography.
Bad idea! Bad idea! Bad idea! Absolutely NOT.
An "AI-driven demonstration cyberattack"? It’s genuinely hard to believe that this is the prescription Noah Smith is pitching!
Consider this first: even before the AI era, major powers like the US, China, and Russia already possessed vast cyberattack capabilities. Don't doubt it for a second—the capability to launch massive cyberattacks is already widespread. Leaving aside high-level espionage like penetrating national defense systems or stealing private bank data, even basic DDoS attacks capable of knocking out airports, power grids, and banking services at scale are not technically difficult. This is common knowledge.
So here is the question: why do these attacks only hit the news sporadically, usually accompanied by vague, mutual accusations of "suspected" state sponsorship? Why haven't they become a daily reality?
It is NOT a question of technical capability! It is precisely because the political consequences of such hostile acts are catastrophic. They are effectively bracketed in the same escalation tier as missile strikes. The severe, mutually assured fallout is the exact reason it hasn't happened yet! This same logic explains why nuclear-armed states haven't launched nuclear strikes—is it because they lack the "capability"?
This is a profoundly dangerous narrative. With the rise of AI, do you really think China isn't aware that these attack capabilities will multiply exponentially? Do they really need the US to "remind" them via a staged breach just to bring them to the AI pacing table?
I think Noah has completely missed the mark here. The AI arms race is strikingly similar to the nuclear arms race. The true terror lies in the unconditional proliferation of apocalyptic capabilities—when 200 nations, or a rogue lab hidden in some dark corner of the world, all gain access to it. The US, China, and the rest of the UN P5 are already terrified of this exact prospect, and that existential threat alone provides more than enough leverage for them to negotiate and cooperate.
Good lord, you do not launch a mock strike on a rival’s strategic defense just to "show them what you can do" and force them to negotiate. That is artificially engineering a modern-day Cuban Missile Crisis—only this time recklessly initiated by the US, assuming Trump were to take Noah's advice seriously!
I strongly recommend Zvi Mowshowitz's blog post analyzing the current moment of AI existential safety going mainstream: https://thezvi.wordpress.com/2026/09/11/jacob-coxon-warns-of-human-extinction-and-triggers-a-preference-cascade/
In particular, he gets into more detail on the question of why this particular moment (triggered by the resignation of a guy nobody had heard of) was when it happened. He also addresses the skeptics who want an explanation of how superintelligent AI would go about killing us, and explains why this question isn't really answerable in a way they'll find satisfying. (Pithiest quote is from Rob Miles: "Look, all I’m asking is that you tell me a specific, detailed story about AI killing everyone, that doesn’t sound to me like science fiction.")