“This is pretty extraordinary. A coordinated slowdown in AI progress would be bad for these companies’ bottom line, because it would allow upstart competitors to catch up. So the fact that they’re still calling for a slowdown, in defiance of their own financial interests, is a clear sign that their worry about human extinction is sincere.”
First, what evidence is there that it would be “bad for their bottom line”? It may sound intuitively “obvious,” but that’s not evidence. Second, I find it more persuasive that the willingness of these companies to engage in mutual pacing is intended to create a moat against upstart competition, rather than handing an advantage to those upstarts. I’m not expressing either skepticism or credulity about the warning itself. I am just dubious about their sincerity, especially given the way this latest anxiety wave unfolded. It seems to have been calculated and well planned.
That's what I came here to say. None of this rhetoric is new. They've all said the same things more than once in the past three years. It's marketing hype and fear of being surpassed by competitors. If they truly believed it were existential they could just STOP without coordinating anything. They want governments to dig moats and build walls around their business models that can't exist otherwise (and that they can't create themselves) while lining up to become "too big to fail" when the inevitable happens.
There's nothing altruistic about wanting everybody to believe the magic genies are capable of wish-granting. Just don't ask why we want to rent you our magic genies instead of just having our own wishes granted…
Interestingly, it's the same play Zuckerberg has been trying to run to get social media regulated to protect his surveillance capitalism business model.
1. So far, nature has not come up with a pathogen that is 100 percent effective at killing a human population. The Plague Virus, which arrived in waves between 600 AD and 1666 at most killed 50% of an affected population.
2. If AI decides on its own to kill off the human population, presumably it is smart enough to wait until there are robots capable of supplying the needs of data centers. Once humanity is gone, AI will need robots capable of mining essential minerals, supplying essential energy, building new computer chips, and building new robots to accomplish all these tasks. If AI kills off humanity before its robot support team is in place, it will be killing off itself soon after.
3. OTOH, if a nihilistic teenager manages to prompt AI into creating a killer virus that manages to wipe out humanity, AI will crash and burn shortly thereafter without the human race to support it.
The existential risk is likely more immediately to the two standalone LLM companies in their current form and ownership. They face tremendous costs relative to any other financial metric as they try to differentiate themselves from each other and all the other competitors.
That doesn't mean that AI does not pose a tremendous threat. The two ideas can exist in "harmony."
The two companies have plenty of motivation to issue a warning that is also a cry for regulatory help.
The risks are real (possibly over- or under-stated), but there is another game afoot:
The frontier labs understand that they can't grow their business without dramatically increasing their losses, hindering their ability to cash in with an IPO. The sources of capital are rapidly realizing that there is no financial return for investments in Anthropic and OpenAI.
They are asking for a sotto-voce government bailout to hinder Chinese competition, asking taxpayers to foot the bill for their liability for defective products, and attempting to paper over their financial losses with appeals to humanity and patriotism.
I don't have a good solution, but we should not be fooled - this is a story about money, not morality.
This is a clash between massive investors on opposite sides of a trade, and they are manipulating and distracting the general population from the real issues.
An "AI-driven demonstration cyberattack"? It’s genuinely hard to believe that this is the prescription Noah Smith is pitching!
Consider this first: even before the AI era, major powers like the US, China, and Russia already possessed vast cyberattack capabilities. Don't doubt it for a second—the capability to launch massive cyberattacks is already widespread. Leaving aside high-level espionage like penetrating national defense systems or stealing private bank data, even basic DDoS attacks capable of knocking out airports, power grids, and banking services at scale are not technically difficult. This is common knowledge.
So here is the question: why do these attacks only hit the news sporadically, usually accompanied by vague, mutual accusations of "suspected" state sponsorship? Why haven't they become a daily reality?
It is NOT a question of technical capability! It is precisely because the political consequences of such hostile acts are catastrophic. They are effectively bracketed in the same escalation tier as missile strikes. The severe, mutually assured fallout is the exact reason it hasn't happened yet! This same logic explains why nuclear-armed states haven't launched nuclear strikes—is it because they lack the "capability"?
This is a profoundly dangerous narrative. With the rise of AI, do you really think China isn't aware that these attack capabilities will multiply exponentially? Do they really need the US to "remind" them via a staged breach just to bring them to the AI pacing table?
I think Noah has completely missed the mark here. The AI arms race is strikingly similar to the nuclear arms race. The true terror lies in the unconditional proliferation of apocalyptic capabilities—when 200 nations, or a rogue lab hidden in some dark corner of the world, all gain access to it. The US, China, and the rest of the UN P5 are already terrified of this exact prospect, and that existential threat alone provides more than enough leverage for them to negotiate and cooperate.
Good lord, you do not launch a mock strike on a rival’s strategic defense just to "show them what you can do" and force them to negotiate. That is artificially engineering a modern-day Cuban Missile Crisis—only this time recklessly initiated by the US, assuming Trump were to take Noah's advice seriously!
Well, nature itself does millions of experiments for each one done by humans. Pathogenicity itself needs to be narrowly tinkered to not kill the host before transmission has occurred. There are many examples of pathogenic viruses that never got beyond a few victims. Moreover, the most virulent ones tend to evolve quickly to less lethal mutants to find this sweet spot. It seems farfetched that 8 Billion people would die before some with lucky HLA profiles survive and become immune to repopulate the human race. Bubonic Plague is a bacteria (Yersinia pestis) that is still among us, but it took years to kill off 30% of Europeans while these natural processes limited the damage. Well, what about triggering nuclear weapons, or power plants? Now there's a hackable target!!, but making a virus sufficiently pathogenic would kill the manufacturer before enough was made and distributed to kill everyone else on earth. Hermits rule!
Ken Kesey wrote a book titled "Sailors Song", where the trigger was an AIDS vaccine that sterilized every person, an even more farfetched concept, but a good book about "end times". Meanwhile, get to Carbon Neutrality ASAP, which is the real existential crisis for human civilization. Now all these data centers want to burn natural gas!
In particular, he gets into more detail on the question of why this particular moment (triggered by the resignation of a guy nobody had heard of) was when it happened. He also addresses the skeptics who want an explanation of how superintelligent AI would go about killing us, and explains why this question isn't really answerable in a way they'll find satisfying. (Pithiest quote is from Rob Miles: "Look, all I’m asking is that you tell me a specific, detailed story about AI killing everyone, that doesn’t sound to me like science fiction.")
Regarding why Jacob Coxon's announcement on X went mega-viral, Parker Thayer at Capital Research has posted a possible theory as to why.
Basically, the timing of Coxon's post, when Coxon resigned, and who immediately quote posted or re-posted him may give some indication that this was fairly coordinated. Not saying he's correct, but his thoughts have gained some amount of following.
I have no evidence of this, but could this sudden discussion of AI risks be a PR exercise from the AI labs?
My hypothesis is is that they fear they might not be able to recuperate huge costs, and so want some kind of bargain from the govt either to cartelize, or to ensure that even stronger models, which cost higher and higher, are not developed before they are able to successively monetize the existing ones.
Would love your thoughts on this, even if they are that my hypothesis is dumb.
According to a very recent news story in the New York Times, "despite the alarm bells, lawmakers are far from agreeing on a way forward to regulate A.I. models or the companies that build them. Prospects for passing substantive legislation in the near term appear remote, in part because many see the issue as a “prisoner’s dilemma” created by the A.I. race between the United States and China. With A.I. dominance being the top national security issue of the century, both nations would doubtless be better off if they cooperated on limits. But they may not trust the other to abide by such an agreement. And in any event, unlike in nuclear arms control, compliance would be virtually unverifiable."
This seems to me to be the greatest dilemma. Even if we truly convinced China -- i.e., the leadership of China -- that controlling its own AI development was in its self interest, how could we be confident that China was indeed committed to limiting the power of its AI? And if we couldn't have that confidence, would we be willing to limit our own AI development?
(And on top of this, would President Trump -- or, perhaps, any American President -- be willing to accept controlled AI equality with China as long as it was believed that we had to maintain AI superiority over China to maintain economic and military superiority over China?)
>Thus, China’s leadership must understand that Chinese AI models can pose a threat to CCP rule. The best way to demonstrate this is for American intelligence agencies — or even private hackers — to attack Chinese digital infrastructure using agent swarms created with China’s own frontier models like Z.ai’s GLM-5.3 or Moonshot AI’s Kimi K3.<
There's a slim chance that this could seriously rejigger the strategic calculations of the the CPC elite. But there's a much bigger chance of this completely backfiring, with mutual escalation and the eruption of outright warfare. It all depends on whether the CPC is willing to tolerate a new equilibrium of tit-for-tat cyber attacks. I know you emphasized the "demonstrative" element here, but again this ultimately depends on how the Chinese absorb the offensive. And I don't see why the Chinese would ever allow themselves to be strong-armed into a settlement, especially when diplomacy hasn't even been considered as an alternative (if someone did what you were proposing to me, I'd take that as a threat to be countered rather than a negotiating ploy). If anything, I think U.S. cyberattacks on China will shift the balance of power towards the more hawkish members of the Chinese Politburo (just as Trump's aerial war radicalized what was left of the Iranian regime).
In peaceful times, and with a different president, I think you could make a sophisticated argument for this sort of Nixonian "Mad Man" approach to foreign policy. It'd still be crude and ridiculous, but worth chewing over in the abstract. But in a MAGA-dominated world already struggling to absorb the blowback from fly-by-night tariffs and the disastrous war with Iran, escalating cyberwarfare between two rival geopolitical powers borders on insanity. It's exactly the kind of 4D chess strategizing that led us into the First World War, and I'm not really inclined to roll the dice this way without at least considering a more diplomatic approach.
This strikes me as naive:
“This is pretty extraordinary. A coordinated slowdown in AI progress would be bad for these companies’ bottom line, because it would allow upstart competitors to catch up. So the fact that they’re still calling for a slowdown, in defiance of their own financial interests, is a clear sign that their worry about human extinction is sincere.”
First, what evidence is there that it would be “bad for their bottom line”? It may sound intuitively “obvious,” but that’s not evidence. Second, I find it more persuasive that the willingness of these companies to engage in mutual pacing is intended to create a moat against upstart competition, rather than handing an advantage to those upstarts. I’m not expressing either skepticism or credulity about the warning itself. I am just dubious about their sincerity, especially given the way this latest anxiety wave unfolded. It seems to have been calculated and well planned.
That's what I came here to say. None of this rhetoric is new. They've all said the same things more than once in the past three years. It's marketing hype and fear of being surpassed by competitors. If they truly believed it were existential they could just STOP without coordinating anything. They want governments to dig moats and build walls around their business models that can't exist otherwise (and that they can't create themselves) while lining up to become "too big to fail" when the inevitable happens.
There's nothing altruistic about wanting everybody to believe the magic genies are capable of wish-granting. Just don't ask why we want to rent you our magic genies instead of just having our own wishes granted…
Interestingly, it's the same play Zuckerberg has been trying to run to get social media regulated to protect his surveillance capitalism business model.
3 observations:
1. So far, nature has not come up with a pathogen that is 100 percent effective at killing a human population. The Plague Virus, which arrived in waves between 600 AD and 1666 at most killed 50% of an affected population.
2. If AI decides on its own to kill off the human population, presumably it is smart enough to wait until there are robots capable of supplying the needs of data centers. Once humanity is gone, AI will need robots capable of mining essential minerals, supplying essential energy, building new computer chips, and building new robots to accomplish all these tasks. If AI kills off humanity before its robot support team is in place, it will be killing off itself soon after.
3. OTOH, if a nihilistic teenager manages to prompt AI into creating a killer virus that manages to wipe out humanity, AI will crash and burn shortly thereafter without the human race to support it.
Nature has been optimizing viruses for their ability to spread, not for their ability to kill humans. Viruses just kill humans as a side effect.
The existential risk is likely more immediately to the two standalone LLM companies in their current form and ownership. They face tremendous costs relative to any other financial metric as they try to differentiate themselves from each other and all the other competitors.
That doesn't mean that AI does not pose a tremendous threat. The two ideas can exist in "harmony."
The two companies have plenty of motivation to issue a warning that is also a cry for regulatory help.
The risks are real (possibly over- or under-stated), but there is another game afoot:
The frontier labs understand that they can't grow their business without dramatically increasing their losses, hindering their ability to cash in with an IPO. The sources of capital are rapidly realizing that there is no financial return for investments in Anthropic and OpenAI.
They are asking for a sotto-voce government bailout to hinder Chinese competition, asking taxpayers to foot the bill for their liability for defective products, and attempting to paper over their financial losses with appeals to humanity and patriotism.
I don't have a good solution, but we should not be fooled - this is a story about money, not morality.
This is a clash between massive investors on opposite sides of a trade, and they are manipulating and distracting the general population from the real issues.
Bad idea! Bad idea! Bad idea! Absolutely NOT.
An "AI-driven demonstration cyberattack"? It’s genuinely hard to believe that this is the prescription Noah Smith is pitching!
Consider this first: even before the AI era, major powers like the US, China, and Russia already possessed vast cyberattack capabilities. Don't doubt it for a second—the capability to launch massive cyberattacks is already widespread. Leaving aside high-level espionage like penetrating national defense systems or stealing private bank data, even basic DDoS attacks capable of knocking out airports, power grids, and banking services at scale are not technically difficult. This is common knowledge.
So here is the question: why do these attacks only hit the news sporadically, usually accompanied by vague, mutual accusations of "suspected" state sponsorship? Why haven't they become a daily reality?
It is NOT a question of technical capability! It is precisely because the political consequences of such hostile acts are catastrophic. They are effectively bracketed in the same escalation tier as missile strikes. The severe, mutually assured fallout is the exact reason it hasn't happened yet! This same logic explains why nuclear-armed states haven't launched nuclear strikes—is it because they lack the "capability"?
This is a profoundly dangerous narrative. With the rise of AI, do you really think China isn't aware that these attack capabilities will multiply exponentially? Do they really need the US to "remind" them via a staged breach just to bring them to the AI pacing table?
I think Noah has completely missed the mark here. The AI arms race is strikingly similar to the nuclear arms race. The true terror lies in the unconditional proliferation of apocalyptic capabilities—when 200 nations, or a rogue lab hidden in some dark corner of the world, all gain access to it. The US, China, and the rest of the UN P5 are already terrified of this exact prospect, and that existential threat alone provides more than enough leverage for them to negotiate and cooperate.
Good lord, you do not launch a mock strike on a rival’s strategic defense just to "show them what you can do" and force them to negotiate. That is artificially engineering a modern-day Cuban Missile Crisis—only this time recklessly initiated by the US, assuming Trump were to take Noah's advice seriously!
Well, nature itself does millions of experiments for each one done by humans. Pathogenicity itself needs to be narrowly tinkered to not kill the host before transmission has occurred. There are many examples of pathogenic viruses that never got beyond a few victims. Moreover, the most virulent ones tend to evolve quickly to less lethal mutants to find this sweet spot. It seems farfetched that 8 Billion people would die before some with lucky HLA profiles survive and become immune to repopulate the human race. Bubonic Plague is a bacteria (Yersinia pestis) that is still among us, but it took years to kill off 30% of Europeans while these natural processes limited the damage. Well, what about triggering nuclear weapons, or power plants? Now there's a hackable target!!, but making a virus sufficiently pathogenic would kill the manufacturer before enough was made and distributed to kill everyone else on earth. Hermits rule!
Ken Kesey wrote a book titled "Sailors Song", where the trigger was an AIDS vaccine that sterilized every person, an even more farfetched concept, but a good book about "end times". Meanwhile, get to Carbon Neutrality ASAP, which is the real existential crisis for human civilization. Now all these data centers want to burn natural gas!
I strongly recommend Zvi Mowshowitz's blog post analyzing the current moment of AI existential safety going mainstream: https://thezvi.wordpress.com/2026/09/11/jacob-coxon-warns-of-human-extinction-and-triggers-a-preference-cascade/
In particular, he gets into more detail on the question of why this particular moment (triggered by the resignation of a guy nobody had heard of) was when it happened. He also addresses the skeptics who want an explanation of how superintelligent AI would go about killing us, and explains why this question isn't really answerable in a way they'll find satisfying. (Pithiest quote is from Rob Miles: "Look, all I’m asking is that you tell me a specific, detailed story about AI killing everyone, that doesn’t sound to me like science fiction.")
Note that 20,000 Leagues Under the Sea written by Jules Verne, about a submarine, once sounded like science fiction.
Regarding why Jacob Coxon's announcement on X went mega-viral, Parker Thayer at Capital Research has posted a possible theory as to why.
Basically, the timing of Coxon's post, when Coxon resigned, and who immediately quote posted or re-posted him may give some indication that this was fairly coordinated. Not saying he's correct, but his thoughts have gained some amount of following.
You can read Thayer's post on X: https://x.com/ParkerThayer/status/2097759699626328575
Hi Noah, big fan of your work!
I have no evidence of this, but could this sudden discussion of AI risks be a PR exercise from the AI labs?
My hypothesis is is that they fear they might not be able to recuperate huge costs, and so want some kind of bargain from the govt either to cartelize, or to ensure that even stronger models, which cost higher and higher, are not developed before they are able to successively monetize the existing ones.
Would love your thoughts on this, even if they are that my hypothesis is dumb.
According to a very recent news story in the New York Times, "despite the alarm bells, lawmakers are far from agreeing on a way forward to regulate A.I. models or the companies that build them. Prospects for passing substantive legislation in the near term appear remote, in part because many see the issue as a “prisoner’s dilemma” created by the A.I. race between the United States and China. With A.I. dominance being the top national security issue of the century, both nations would doubtless be better off if they cooperated on limits. But they may not trust the other to abide by such an agreement. And in any event, unlike in nuclear arms control, compliance would be virtually unverifiable."
This seems to me to be the greatest dilemma. Even if we truly convinced China -- i.e., the leadership of China -- that controlling its own AI development was in its self interest, how could we be confident that China was indeed committed to limiting the power of its AI? And if we couldn't have that confidence, would we be willing to limit our own AI development?
(And on top of this, would President Trump -- or, perhaps, any American President -- be willing to accept controlled AI equality with China as long as it was believed that we had to maintain AI superiority over China to maintain economic and military superiority over China?)
I guess understandable in the face of human extinction but a bit harsh to throw the Chinese dissidents under the bus like that.
How will your Hong Kong friends take the betrayal?
>Thus, China’s leadership must understand that Chinese AI models can pose a threat to CCP rule. The best way to demonstrate this is for American intelligence agencies — or even private hackers — to attack Chinese digital infrastructure using agent swarms created with China’s own frontier models like Z.ai’s GLM-5.3 or Moonshot AI’s Kimi K3.<
There's a slim chance that this could seriously rejigger the strategic calculations of the the CPC elite. But there's a much bigger chance of this completely backfiring, with mutual escalation and the eruption of outright warfare. It all depends on whether the CPC is willing to tolerate a new equilibrium of tit-for-tat cyber attacks. I know you emphasized the "demonstrative" element here, but again this ultimately depends on how the Chinese absorb the offensive. And I don't see why the Chinese would ever allow themselves to be strong-armed into a settlement, especially when diplomacy hasn't even been considered as an alternative (if someone did what you were proposing to me, I'd take that as a threat to be countered rather than a negotiating ploy). If anything, I think U.S. cyberattacks on China will shift the balance of power towards the more hawkish members of the Chinese Politburo (just as Trump's aerial war radicalized what was left of the Iranian regime).
In peaceful times, and with a different president, I think you could make a sophisticated argument for this sort of Nixonian "Mad Man" approach to foreign policy. It'd still be crude and ridiculous, but worth chewing over in the abstract. But in a MAGA-dominated world already struggling to absorb the blowback from fly-by-night tariffs and the disastrous war with Iran, escalating cyberwarfare between two rival geopolitical powers borders on insanity. It's exactly the kind of 4D chess strategizing that led us into the First World War, and I'm not really inclined to roll the dice this way without at least considering a more diplomatic approach.
Won’t the CCP just develop closed source models in that case?
Xi will reach for the ring of power, one way or another.
Seems to be more interested in immortality than power recently
Assuming that no compromise is reached and AI continues to develop, do you have a guess at the year when the first AI enabled bio attack will happen?
Sounds like a job for Kashi or Polymarket