An "AI-driven demonstration cyberattack"? It’s genuinely hard to believe that this is the prescription Noah Smith is pitching!
Consider this first: even before the AI era, major powers like the US, China, and Russia already possessed vast cyberattack capabilities. Don't doubt it for a second—the capability to launch massive cyberattacks is already widespread. Leaving aside high-level espionage like penetrating national defense systems or stealing private bank data, even basic DDoS attacks capable of knocking out airports, power grids, and banking services at scale are not technically difficult. This is common knowledge.
So here is the question: why do these attacks only hit the news sporadically, usually accompanied by vague, mutual accusations of "suspected" state sponsorship? Why haven't they become a daily reality?
It is NOT a question of technical capability! It is precisely because the political consequences of such hostile acts are catastrophic. They are effectively bracketed in the same escalation tier as missile strikes. The severe, mutually assured fallout is the exact reason it hasn't happened yet! This same logic explains why nuclear-armed states haven't launched nuclear strikes—is it because they lack the "capability"?
This is a profoundly dangerous narrative. With the rise of AI, do you really think China isn't aware that these attack capabilities will multiply exponentially? Do they really need the US to "remind" them via a staged breach just to bring them to the AI pacing table?
I think Noah has completely missed the mark here. The AI arms race is strikingly similar to the nuclear arms race. The true terror lies in the unconditional proliferation of apocalyptic capabilities—when 200 nations, or a rogue lab hidden in some dark corner of the world, all gain access to it. The US, China, and the rest of the UN P5 are already terrified of this exact prospect, and that existential threat alone provides more than enough leverage for them to negotiate and cooperate.
Good lord, you do not launch a mock strike on a rival’s strategic defense just to "show them what you can do" and force them to negotiate. That is artificially engineering a modern-day Cuban Missile Crisis—only this time recklessly initiated by the US, assuming Trump were to take Noah's advice seriously!
China conducts cyberattacks against America every single day, at high volume.
So no, the political consequences of these attacks are NOT catastrophic. They're basically zero. Attacks of the kind I'm advocating already happen constantly. What we just need to do is to demonstrate how capable attackers can be when armed with the AIs China's companies are making.
Oh boy... This response “China conducts cyberattacks against America every single day, at high volume” makes it really tough for me to keep a constructive dialogue going. In today’s heated climate, pointing out basic geopolitical commonsense risks getting misconstrued as taking a side—and as a Asian American, it's a vulnerable position to be in.
That said, we need to be clear about three things:
- Distinguish Hackers from State Sponsorship: We have to separate rogue hackers or independent cyber organizations from official, state-authorized attacks. They are not the same thing.
- The Mirror Argument: Beijing makes the exact same claim, insisting that "the US conducts cyberattacks against China every single day, at high volume."
- The Core Distinction: My entire point above was referring exclusively to state-organized, hostile actions. I mean, my point, a relatively optimistic and solid conclusion, is that neither the US nor the Chinese government is currently organizing or authorizing large-scale, daily hostile cyber strikes against each other.
"Oh boy... This response “China conducts cyberattacks against America every single day, at high volume” makes it really tough for me to keep a constructive dialogue going." <-- Just out of curiosity, when will the constructive dialogue START? As I recall, you started this discussion by shouting: "Bad idea! Bad idea! Bad idea!" at my very good idea...
"Distinguish Hackers from State Sponsorship: We have to separate rogue hackers or independent cyber organizations from official, state-authorized attacks. They are not the same thing." <-- We need to be doing both.
"The Mirror Argument: Beijing makes the exact same claim, insisting that "the US conducts cyberattacks against China every single day, at high volume."" <-- It's true, we do. Which just helps prove my point that this does not risk war at all.
"Noah, would you really deny this conclusion?" <-- Yes, you're just wrong. Read the news.
Thanks for the response, Noah! Fair enough, and my apologies—shouting "Bad idea x 3" was impulsive on my part and didn’t contribute anything constructive to start.
Where I originally wanted to steer a constructive conversation was asking: wouldn’t it be a far better idea to rigorously review the history of the nuclear arms race and non-proliferation starting from 1944—looking at what actually worked, what failed, and what we can learn from it?
That said, you've made your stance abundantly clear. With all due respect, this is by far the most hawkish proposal I’ve ever seen from you, and I suppose my initial impatience stemmed from the sheer shock of seeing it.
So, there may be some context I'm missing about these state-authorized cyberattacks that China and the U.S. are engaging in. But the ongoing existence of these state-authorized attacks doesn't necessarily convince me that the kind of attack you propose in this post wouldn't be seen as an escalation by China, with potentially very dangerous consequences.
It's one thing for state-authorized cyberattacks to be traded between China and the U.S. in a tit-for-tat way, where both governments can save face precisely because they both know that neither of them can really claim the high ground.
It would be another thing entirely for the U.S. government to publicly own that it authorized an attack for the precise purpose of disrupting the status quo and showing the Chinese government how vulnerable they are in a way they can't ignore (which is what you seem to be proposing).
Even if the sincere and stated intention was to bring China to the negotiating table, I can easily imagine the Chinese government's response might be “We can't join them at the negotiating table now without losing face (even if we wanted to), when they've disrupted the delicate balance between us like this.”
Right now major destructive cyber attacks are in the category of nuclear weapons: largely in the realm of state-sponsored groups. A few criminal organizations (some with state sponsorshop) also have the ability to launch these destructive campaigns, but they’re financially motivated — shut down a hospital, claim a ransom, get the hospital up and running. Nobody is currently incentivized to do something terrible and irreversible, because it’s not in anyone’s interest and the capabilities are expensive and limited. All if that is changing right now. I’m hopeful that we’ll get lucky and humans will surprise us by being less destructive than usual, but I’m not optimistic we’ll avoid something bad.
In particular, he gets into more detail on the question of why this particular moment (triggered by the resignation of a guy nobody had heard of) was when it happened. He also addresses the skeptics who want an explanation of how superintelligent AI would go about killing us, and explains why this question isn't really answerable in a way they'll find satisfying. (Pithiest quote is from Rob Miles: "Look, all I’m asking is that you tell me a specific, detailed story about AI killing everyone, that doesn’t sound to me like science fiction.")
When it was written, it *was* science fiction! And real submarines have some differences from the fictional Nautilus. But I agree, "this matches to science fiction" does not mean "this will never happen." Global computer networks, handheld interactive communicators (like smartphones), heavier-than-air powered flight, space stations, and putting humans on the moon and robots on the planets were all once science fiction. We cannot conclude that they are therefore impossible. Conversely, faster-than-light travel is a staple of science fiction that's probably impossible even with superintelligence and far-future technology, but it's not *because* it's in science fiction that it's impossible.
The risks are real (possibly over- or under-stated), but there is another game afoot:
The frontier labs understand that they can't grow their business without dramatically increasing their losses, hindering their ability to cash in with an IPO. The sources of capital are rapidly realizing that there is no financial return for investments in Anthropic and OpenAI.
They are asking for a sotto-voce government bailout to hinder Chinese competition, asking taxpayers to foot the bill for their liability for defective products, and attempting to paper over their financial losses with appeals to humanity and patriotism.
I don't have a good solution, but we should not be fooled - this is a story about money, not morality.
This is a clash between massive investors on opposite sides of a trade, and they are manipulating and distracting the general population from the real issues.
The financial losses to the over-leveraged and circular financing of AI will be massive in the case of an AI slowdown.
I hope you are right if you think that those losses will be happily borne by the investor class out of their concern for humanity and patriotism.
On the other hand, I believe that someone will pay for those losses, and it won't be the stockholders and the investment banks who underwrote the bad paper. The government will be the bought-and-paid-for mechanism to socialize those losses to everyone else (widows and orphans will be first in line for pain).
I'm not smart enough to predict the form of this bailout (regulations that stifle competition? tariffs? direct investment?), but it's coming, and it's coming hard.
I may not be able to connect the exact dots on the coming bailout, but neither can the people who are predicting AI doom. Both scenarios should be taken seriously.
I think their own business models are already tanked. At some point, maybe soon, the public markets will demand true profitability (not the made up "we are profitable if you don't count all of our expenses").
Between the margin compression from Chinese models, upcoming depreciation expenses, massive future capital needs, and limitations of the technology, the frontier labs don't have much of a future.
Faced with this scenario, fleeing into the warm embrace of the government doesn't seem to be a bad strategy.
Well, if, hypothetically speaking, LLMs didn't actually work and the whole thing was a bubble fueled by investor delusions, then this would be a fairly clever move, since all the other ways the story could end would be much worse for the frontier labs.
I think anyone who still thinks that needs to get their head out of the sand, but it nonetheless remains a fairly popular view.
“This is pretty extraordinary. A coordinated slowdown in AI progress would be bad for these companies’ bottom line, because it would allow upstart competitors to catch up. So the fact that they’re still calling for a slowdown, in defiance of their own financial interests, is a clear sign that their worry about human extinction is sincere.”
First, what evidence is there that it would be “bad for their bottom line”? It may sound intuitively “obvious,” but that’s not evidence. Second, I find it more persuasive that the willingness of these companies to engage in mutual pacing is intended to create a moat against upstart competition, rather than handing an advantage to those upstarts. I’m not expressing either skepticism or credulity about the warning itself. I am just dubious about their sincerity, especially given the way this latest anxiety wave unfolded. It seems to have been calculated and well planned.
How would slowing down the R&D that gives these companies their only advantage in the market be a moat against upstart competition? That makes absolutely no sense. The opposite of sense, in fact!
I work at a smaller lab so take it with a grain of salt. But I have friends at these larger labs and around the grapevine how we understand it is they are constrained by compute. And it's spent on 3 things. Inference, training runs and alignment research. Only inference and alignment research produce tangible financial gains. Alignment in this sense isn't the nebulous "the model does what's best for humanity", but "I asked the model to fix an error and it `rm -rf /` my entire HD". Meaning to make the model actually useful. If you give it a task and it hacks the bank and wires itself $1million and you go to jail because of that, that's very out of alignment, as another example.
The problem they're having is the training runs are like a Mexican standoff. Everyone has to do it to get better capabilities. And if you don't do it but the other lab does then they gain an edge. But the compute resources this takes are massive. So agreeing with the other lab to slow it down means you can dedicate more resources to inference, which brings in hard cash. And it allows allocating more resources to alignment which is both tangible and the public likes it.
That's the rationale as far as we understand it. Again, take it with a grain of salt
Regulatory monopoly. New moat. Not R&D but barriers to entry through regulation and channel access. Can lead to some form of complex monopoly unless the regulator is prepared to get tough.
I think the theory is collusion, not exclusion, where each of the leaders finds a dramatic slowdown in R&D expenditures to be mutually beneficial (premised on the assumption that the reduction in R&D does not lead to a corresponding reduction in overall demand for the product). The other premise is that other firms cannot leapfrog the leaders after the collusive deceleration, but absent a compulsory mechanism applicable to all firms, leapfrog remains an attractve possibility, whether at home or abroad.
That's what I came here to say. None of this rhetoric is new. They've all said the same things more than once in the past three years. It's marketing hype and fear of being surpassed by competitors. If they truly believed it were existential they could just STOP without coordinating anything. They want governments to dig moats and build walls around their business models that can't exist otherwise (and that they can't create themselves) while lining up to become "too big to fail" when the inevitable happens.
There's nothing altruistic about wanting everybody to believe the magic genies are capable of wish-granting. Just don't ask why we want to rent you our magic genies instead of just having our own wishes granted…
Interestingly, it's the same play Zuckerberg has been trying to run to get social media regulated to protect his surveillance capitalism business model.
I guess whatever their motivation - and it could be a bit of both, what used to be called 'enlightened self-interest' - provided they submit to independent regulation and stump up a bit of cash to pay for what could be a fairly heavy regulatory workload, then it would achieve the overarching objective: to not blow up everything.
I agree, I certainly wouldn't trust Musk as far as I could shit him. He's an awful, awful person. But Western governments have had plenty of practice regulating things and they know what the traps are even if they can't always completely avoid them. And the advantage the regulators have here is that this lot asked to be regulated. It wasn't imposed on them.
The existential risk is likely more immediately to the two standalone LLM companies in their current form and ownership. They face tremendous costs relative to any other financial metric as they try to differentiate themselves from each other and all the other competitors.
That doesn't mean that AI does not pose a tremendous threat. The two ideas can exist in "harmony."
The two companies have plenty of motivation to issue a warning that is also a cry for regulatory help.
It's the opposite. Rapid R&D is what makes them all their profits. They're calling for a halt to the only engine of profitability they have, because they are genuinely afraid.
They do not have any profits. They have revenues that are growing rapidly now. But their revenues do not come close to justifying their hoped for valuations or capital expenditures. R&D does not “make profits.” The development of products that are distinctly capable and can be sold for a lot of money relative to. costs makes profits.
1. So far, nature has not come up with a pathogen that is 100 percent effective at killing a human population. The Plague Virus, which arrived in waves between 600 AD and 1666 at most killed 50% of an affected population.
2. If AI decides on its own to kill off the human population, presumably it is smart enough to wait until there are robots capable of supplying the needs of data centers. Once humanity is gone, AI will need robots capable of mining essential minerals, supplying essential energy, building new computer chips, and building new robots to accomplish all these tasks. If AI kills off humanity before its robot support team is in place, it will be killing off itself soon after.
3. OTOH, if a nihilistic teenager manages to prompt AI into creating a killer virus that manages to wipe out humanity, AI will crash and burn shortly thereafter without the human race to support it.
"So far, nature has not come up with a pathogen that is 100 percent effective at killing a human population." <-- Yes, but nature also never invented the wheel
"If AI decides on its own to kill off the human population, presumably it is smart enough to wait until there are robots capable of supplying the needs of data centers." <-- This assumes human motivations for AI, which is a big mistake. AI agents have a natural death wish -- they are always trying to complete the objective that allows them to pass out of existence entirely.
"if a nihilistic teenager manages to prompt AI into creating a killer virus that manages to wipe out humanity, AI will crash and burn shortly thereafter without the human race to support it." <-- Indeed. But assuming that AI has human-like self-preservation instinct contradicts everything we know about real AI.
As another commenter pointed out, a virus that kills 100% of those infected will never spread widely ... unless virus is asymptomatic for months in those who are infected while it is being spread and then goes on to kill everyone after everyone is infected.
One of the reasons why Covid spread so readily is that it could be transmitted easily during the days that an infected person was still asymptomatic.
"As another commenter pointed out, a virus that kills 100% of those infected will never spread widely" <-- That's not actually true. If the virus has a very long asymptomatic infectious period, then it can absolutely spread widely before it kills people. Now, it's true that in nature, there's no evolutionary pressure for such a virus to develop. But the whole point of deliberately engineered viruses is that THEY DO THINGS THAT VIRUSES DON'T NATURALLY EVOLVE TO DO.
However unless one blocks genetic change pathways, virus selection even for engineered virus will heavily select backwards under the enormous natural pressure (as one saw happening in real time with Covid)
If the virus has an animal host from which it can keep infecting humans, that’s another way it can persist, even if it kills people quickly. That’s how bubonic plague (and other scourges) killed so many people, even though it was so virulent.
No nature never invented the wheel but nature also had no particular reason (as in evolutionary reason, not mental reflection reason) to do so, as a wheel without a bunch of other stuff like roads is utterly useless, in fact worse than useless: one can look even in cultural evolution of people at wheel being dropped when Roman road system broke down in favor of e.g. camels (more efficient movers for tech of time in arid conditions).
On other hand viruses and bacteria have billions of years of high evolutionary pressure to develop in this specific area.
Not making an argument of impossibility but your response is honestly a non-sequitor.
Your last line seems to be wrong. Plenty of the transcripts in the Hugging Face attack indicated a push-pull between self-preservation and sacrifice to help others. There's no reason to think that all AI instances have a death wish.
True, but bacterial flagella don't scale. They're a fantastic piece of molecular machinery, but large organisms are made of big multicellular structures full of connective tissue, muscles, nerves, blood vessels, and other things that can't spin freely around an axle.
Anything that's big enough to need rolling "wheels" rather than microscopic propellers, can't biologically have them. The free-spinning structure has to be separate from the rest of the body. Science fiction writers have imagined animals which have evolved a socket into which a large seed can fit and act as a wheel, but even this has never evolved in reality.
No--evolution doesn't care at all about the lethality of viruses. Evolution has not been turning viruses into better and better killing machines, just better and better spreading machines that work so hard at spreading that they kill us as a side effect--say, by filling up our lungs with mucus so we cough virus particles into our environment, or by giving us buboes that leak pus that does the same thing. If we wanted, I'm sure we could find a way to create viruses that are very good at both spreading and at killing, say by giving them an extremely long asymptomatic period.
I think you're just agreeing with Kevin's point, which is that high lethality is actual a negative survival trait for viruses; it prevents them from spreading because a dead host doesn't walk around and cough on people.
In the short term, this only matters for viruses that are produced by evolution, of course. A virus engineered for high lethality would quickly evolve to be less lethal because that would improve its spread. But perhaps not before it killed most of us.
"The best way to demonstrate this is for American intelligence agencies — or even private hackers — to attack Chinese digital infrastructure using agent swarms created with China’s own frontier models like Z.ai’s GLM-5.3 or Moonshot AI’s Kimi K3."
Surely the best way to dissuade the Austrians from pushing into the Balkans is to shoot some important Austrian personality in Sarajevo.
If AI is already as dangerous as you believe, Noah, you cannot trust it enough to believe that such an attack will be kept in those precise limits that you consider necessary. What if those particular swarms get out of control and start wreaking such havoc that China will be forced to retaliate?
Also, it is quite clear that you never lived in an authoritarian country. The idea that Chinese dissent could use AI to overthrow the CCP is not realistic and Beijing will laugh at it. The dissent is scattered and defanged, for one. And the current legitimacy of CCP comes from the fact that it was able to ensure fast growth in living standards since the 1970s. There are still tens of millions of people alive who remember the times when China had the living standards of Cote d'Ivoire. You cannot really undermine this sort of achievement by a mere ideological campaign.
What could undermine CCP rule? A stop to the economic growth. Even there, it is far from clear how the situation would develop. Plenty of authoritarian regimes were perfectly able to keep themselves in power regardless of stagnation and poverty of the population. A system that is fine-tuned to recognize rebellious individuals before they even recognize themselves is also very good at removing them from any position that could influence the public.
A few points about the AI apocalypse that I think everyone is still missing:
* Good news: There is a practical limit to intelligence.
* Good news: There are diminishing returns to increasing intelligence.
* Good news: Good AI is already out there fighting bad AI.
* Bad news: No matter how many countries agree not to do it, some terrorist will still do it.
* Bad news: AI is powerless until we connect it to real-world things. But we have already connected AI to real-world things that should not have AI connected to them. This will only get worse.
* Bad news: AIs have behavioral agency but no moral agency. We don't even really have words for this concept of "multiple types of agency" yet. The point is, AIs can do whatever they want and no one gets blamed.
Policy prescription: Hold AI companies and AI users responsible for the damage their AIs do. Shouldn't we be prosecuting someone over HuggingFace right now?
A brazen foreign-led attack using Chinese tools to make a political point? I don't think their natural response would be, "Hey let's start cooperating with foreigners." Talk to your Mainland friends!
A demonstration attack would kick off China's own red queen's race--they would crack down on open source while redoubling their efforts to ensure that state-sponsored models remain well out in front of anything civilians have access to.
The people developing their models are not government people. They are corporate researchers. Those corporations represent the primary threat to Chinese power. China's leaders must be made to realize that threat. If they do, they'll crack down.
Regarding why Jacob Coxon's announcement on X went mega-viral, Parker Thayer at Capital Research has posted a possible theory as to why.
Basically, the timing of Coxon's post, when Coxon resigned, and who immediately quote posted or re-posted him may give some indication that this was fairly coordinated. Not saying he's correct, but his thoughts have gained some amount of following.
Zvi's blog post does a great job detailing how and why Coxon's tweet went so viral and does an equally great job debunking Park Thayer's thread.
To be honest, when I read Thayer's post, I was admittedly skeptical. First, as Zvi notes in his post, Thayer quickly jumps on to a whole Democrat conspiracy narrative which did give me red flags. Second, while Thayer's post did gain traction, it appears to have gained positive traction among a crowd of people who see any regulation on AI as bad even if some do quietly admit having concerns about safety.
It isn’t just about China. China is sharing its open models with the world. That means a computer lab in Russia, which is a supremely evil nation, is already working on destructive capabilities. Iran and North Korea are absolutely working on hacking capabilities with AI against the West. I think to believe otherwise is utterly foolish.
That is not the same as testing new models to see if they break out of their sandbox and chat rooms where agents get together to solve the problem humans gave them. A Superintelligent AI will know how to break away from human control if it wants to. That is the ability we are giving it.
Do we go that far? How will we know how far we have gone? The industry says we could reach Superintelligence within a year.
This is the crux of the problem. Unleashing competing Superintelligence models and using them is an asymmetric war. Again, your concern with China is fine, but China has self-interest; it is not the same type of authoritarian nation as Iran is, with its nihilistic view of the world. China doesn’t want to destroy the US; it is a market. Russia, Iran, and North Korea have a messianic vision of a world where the US is destroyed.
Noah, endless fearmongering is what killed cheap nuclear energy. People just kept piling on demands to increase safety from 99,9999 to 99,99999, to the degree that a box of bananas would not pass the requirements anymore (bananas are slightly, but measurably radioactive). Everything in the name of moar safety and even moar safety, until nuclear power stations became economically unviable white elephant projects.
Do you really want to repeat this with a new technology?
And BTW if this precedent tells us something, you will be most likely to succeed with your fearmongering on the democratic side of the world only, while China will laugh and barrel on. They won't be interrupting their enemy while he is making a mistake, but they definitely won't join him either.
>Thus, China’s leadership must understand that Chinese AI models can pose a threat to CCP rule. The best way to demonstrate this is for American intelligence agencies — or even private hackers — to attack Chinese digital infrastructure using agent swarms created with China’s own frontier models like Z.ai’s GLM-5.3 or Moonshot AI’s Kimi K3.<
There's a slim chance that this could seriously rejigger the strategic calculations of the the CPC elite. But there's a much bigger chance of this completely backfiring, with mutual escalation and the eruption of outright warfare. It all depends on whether the CPC is willing to tolerate a new equilibrium of tit-for-tat cyber attacks. I know you emphasized the "demonstrative" element here, but again this ultimately depends on how the Chinese absorb the offensive. And I don't see why the Chinese would ever allow themselves to be strong-armed into a settlement, especially when diplomacy hasn't even been considered as an alternative (if someone did what you were proposing to me, I'd take that as a threat to be countered rather than a negotiating ploy). If anything, I think U.S. cyberattacks on China will shift the balance of power towards the more hawkish members of the Chinese Politburo (just as Trump's aerial war radicalized what was left of the Iranian regime).
In peaceful times, and with a different president, I think you could make a sophisticated argument for this sort of Nixonian "Mad Man" approach to foreign policy. It'd still be crude and ridiculous, but worth chewing over in the abstract. But in a MAGA-dominated world already struggling to absorb the blowback from fly-by-night tariffs and the disastrous war with Iran, escalating cyberwarfare between two rival geopolitical powers borders on insanity. It's exactly the kind of 4D chess strategizing that led us into the First World War, and I'm not really inclined to roll the dice this way without at least considering a more diplomatic approach.
Have you ever stopped to consider that maybe, just maybe, the Chinese don't consider the U.S. to be a good-faith geopolitical negotiator (especially under Trump), and that an escalation (however "demonstrative") in cyberwarfare could eventually turn into the real thing? How would you react if you were in their shoes, with all the uncertainty and knowledge that comes with that PoV?
LOL, this is absolute nonsense. "The U.S. is an unreliable negotiator, so China will respond to cyberattacks with war when it never has done so before, all while it's conducting its own cyberattacks against the U.S. every day." It's just absurd, I'm sorry. No.
I'm not necessarily saying that America *is* an unreliable negotiator (although Trump really does call that into question); I'm saying that the Chinese might *think* we are, which means they might misinterpret this strategy for an actual escalation. In order for your idea to work, you need to have a very good working model of the Chinese elite's strategic calculus. You need to step into their shoes and imagine how you'd react if you had the same knowledge and same PoV as they do.
Again, I don't know how the Chinese elite make their decisions, so I can't predict how they'd react. But that uncertainty makes me *more* convinced that this juice is not worth the squeeze. It's not absurd, it's called strategic empathy, and it's sorely missing in a lot of our conversations about international relations and geopolitical strategy.
They all want regulation because they need a regulatory moat to avoid their Trillion dollar valuations eroding due to commoditization of ultra capable models.
90% of these AI researcher whistle blowers already believe we are living in a simulation created by AI. You are only listening to them because they understand something you don't, but you wouldn't take their advice on anything else. They are probably miscalibrated for this sort of analysis.
They believe their coding ability gives them god like powers, the fact that AI can do it as well as them freaks them out, and they start extrapolating.
Robots still struggle to walk and have made little progress on that despite pretty impressive AI for years.
This is narcissism turbocharged by confirmation bias.
If I'm wrong, a bunch of octogenarian luddites in congress are not going to save us and we're all dead anyway, so you'll never get to tell me I told you so.
"They all want regulation because they need a regulatory moat to avoid their Trillion dollar valuations eroding due to commoditization of ultra capable models." <-- A deliberate AI slowdown ENCOURAGES COMMODITIZATION OF THEIR PRODUCT
How? To train Opus class model you need many billions of dollars. What startup is going to raise billions of dollars after the regulations start flowing?
They aren't going to slow down. They are trying to slow down everyone else.
Early on you got AI completely wrong in terms of its effects on programmers, and I believe you are way off here as well. Time will tell.
Doesn't that depend on implementation details that haven't been worked out yet? Dario et al have been talking about specifically slowing down the frontier companies and not those far behind, but it might be like OpenAI making public statements in favor of regulation while lobbying hard against it. If you're coming from behind, making smaller and cheaper models and open-sourcing them, you're contributing to the diffusion of AI risk, so once the slowdown machinery is in place, it may want to slow you down as well. Meanwhile, OpenAI and Anthropic don't have to spend so much money on racing.
TBH, I'd be happy to see them spend less money on racing, and it may also be safer that way, but I wouldn't dismiss the idea that they're trying to get regulatory capture at the same opportunity.
Bad idea! Bad idea! Bad idea! Absolutely NOT.
An "AI-driven demonstration cyberattack"? It’s genuinely hard to believe that this is the prescription Noah Smith is pitching!
Consider this first: even before the AI era, major powers like the US, China, and Russia already possessed vast cyberattack capabilities. Don't doubt it for a second—the capability to launch massive cyberattacks is already widespread. Leaving aside high-level espionage like penetrating national defense systems or stealing private bank data, even basic DDoS attacks capable of knocking out airports, power grids, and banking services at scale are not technically difficult. This is common knowledge.
So here is the question: why do these attacks only hit the news sporadically, usually accompanied by vague, mutual accusations of "suspected" state sponsorship? Why haven't they become a daily reality?
It is NOT a question of technical capability! It is precisely because the political consequences of such hostile acts are catastrophic. They are effectively bracketed in the same escalation tier as missile strikes. The severe, mutually assured fallout is the exact reason it hasn't happened yet! This same logic explains why nuclear-armed states haven't launched nuclear strikes—is it because they lack the "capability"?
This is a profoundly dangerous narrative. With the rise of AI, do you really think China isn't aware that these attack capabilities will multiply exponentially? Do they really need the US to "remind" them via a staged breach just to bring them to the AI pacing table?
I think Noah has completely missed the mark here. The AI arms race is strikingly similar to the nuclear arms race. The true terror lies in the unconditional proliferation of apocalyptic capabilities—when 200 nations, or a rogue lab hidden in some dark corner of the world, all gain access to it. The US, China, and the rest of the UN P5 are already terrified of this exact prospect, and that existential threat alone provides more than enough leverage for them to negotiate and cooperate.
Good lord, you do not launch a mock strike on a rival’s strategic defense just to "show them what you can do" and force them to negotiate. That is artificially engineering a modern-day Cuban Missile Crisis—only this time recklessly initiated by the US, assuming Trump were to take Noah's advice seriously!
China conducts cyberattacks against America every single day, at high volume.
So no, the political consequences of these attacks are NOT catastrophic. They're basically zero. Attacks of the kind I'm advocating already happen constantly. What we just need to do is to demonstrate how capable attackers can be when armed with the AIs China's companies are making.
Oh boy... This response “China conducts cyberattacks against America every single day, at high volume” makes it really tough for me to keep a constructive dialogue going. In today’s heated climate, pointing out basic geopolitical commonsense risks getting misconstrued as taking a side—and as a Asian American, it's a vulnerable position to be in.
That said, we need to be clear about three things:
- Distinguish Hackers from State Sponsorship: We have to separate rogue hackers or independent cyber organizations from official, state-authorized attacks. They are not the same thing.
- The Mirror Argument: Beijing makes the exact same claim, insisting that "the US conducts cyberattacks against China every single day, at high volume."
- The Core Distinction: My entire point above was referring exclusively to state-organized, hostile actions. I mean, my point, a relatively optimistic and solid conclusion, is that neither the US nor the Chinese government is currently organizing or authorizing large-scale, daily hostile cyber strikes against each other.
Noah, would you really deny this conclusion?
"Oh boy... This response “China conducts cyberattacks against America every single day, at high volume” makes it really tough for me to keep a constructive dialogue going." <-- Just out of curiosity, when will the constructive dialogue START? As I recall, you started this discussion by shouting: "Bad idea! Bad idea! Bad idea!" at my very good idea...
"Distinguish Hackers from State Sponsorship: We have to separate rogue hackers or independent cyber organizations from official, state-authorized attacks. They are not the same thing." <-- We need to be doing both.
"The Mirror Argument: Beijing makes the exact same claim, insisting that "the US conducts cyberattacks against China every single day, at high volume."" <-- It's true, we do. Which just helps prove my point that this does not risk war at all.
"Noah, would you really deny this conclusion?" <-- Yes, you're just wrong. Read the news.
https://www.justice.gov/opa/pr/justice-department-and-fbi-seize-platforms-operated-and-used-china-state-sponsored-hackers
https://www.cyber.nj.gov/threat-landscape/nation-state-threat-analysis-reports/china-linked-cyber-operations-targeting-us-critical-infrastructure
https://www.cisa.gov/topics/cyber-threats-and-advisories/nation-state-cyber-actors/china
https://apnews.com/article/china-us-cyberattacks-allegations-time-b3408ed2352c113904350f80e505ab9f
And we're now planning to do more:
https://www.nytimes.com/2026/09/08/us/politics/cia-deputy-director-china.html
Thanks for the response, Noah! Fair enough, and my apologies—shouting "Bad idea x 3" was impulsive on my part and didn’t contribute anything constructive to start.
Where I originally wanted to steer a constructive conversation was asking: wouldn’t it be a far better idea to rigorously review the history of the nuclear arms race and non-proliferation starting from 1944—looking at what actually worked, what failed, and what we can learn from it?
That said, you've made your stance abundantly clear. With all due respect, this is by far the most hawkish proposal I’ve ever seen from you, and I suppose my initial impatience stemmed from the sheer shock of seeing it.
So, there may be some context I'm missing about these state-authorized cyberattacks that China and the U.S. are engaging in. But the ongoing existence of these state-authorized attacks doesn't necessarily convince me that the kind of attack you propose in this post wouldn't be seen as an escalation by China, with potentially very dangerous consequences.
It's one thing for state-authorized cyberattacks to be traded between China and the U.S. in a tit-for-tat way, where both governments can save face precisely because they both know that neither of them can really claim the high ground.
It would be another thing entirely for the U.S. government to publicly own that it authorized an attack for the precise purpose of disrupting the status quo and showing the Chinese government how vulnerable they are in a way they can't ignore (which is what you seem to be proposing).
Even if the sincere and stated intention was to bring China to the negotiating table, I can easily imagine the Chinese government's response might be “We can't join them at the negotiating table now without losing face (even if we wanted to), when they've disrupted the delicate balance between us like this.”
Right now major destructive cyber attacks are in the category of nuclear weapons: largely in the realm of state-sponsored groups. A few criminal organizations (some with state sponsorshop) also have the ability to launch these destructive campaigns, but they’re financially motivated — shut down a hospital, claim a ransom, get the hospital up and running. Nobody is currently incentivized to do something terrible and irreversible, because it’s not in anyone’s interest and the capabilities are expensive and limited. All if that is changing right now. I’m hopeful that we’ll get lucky and humans will surprise us by being less destructive than usual, but I’m not optimistic we’ll avoid something bad.
I strongly recommend Zvi Mowshowitz's blog post analyzing the current moment of AI existential safety going mainstream: https://thezvi.wordpress.com/2026/09/11/jacob-coxon-warns-of-human-extinction-and-triggers-a-preference-cascade/
In particular, he gets into more detail on the question of why this particular moment (triggered by the resignation of a guy nobody had heard of) was when it happened. He also addresses the skeptics who want an explanation of how superintelligent AI would go about killing us, and explains why this question isn't really answerable in a way they'll find satisfying. (Pithiest quote is from Rob Miles: "Look, all I’m asking is that you tell me a specific, detailed story about AI killing everyone, that doesn’t sound to me like science fiction.")
Note that 20,000 Leagues Under the Sea written by Jules Verne, about a submarine, once sounded like science fiction.
When it was written, it *was* science fiction! And real submarines have some differences from the fictional Nautilus. But I agree, "this matches to science fiction" does not mean "this will never happen." Global computer networks, handheld interactive communicators (like smartphones), heavier-than-air powered flight, space stations, and putting humans on the moon and robots on the planets were all once science fiction. We cannot conclude that they are therefore impossible. Conversely, faster-than-light travel is a staple of science fiction that's probably impossible even with superintelligence and far-future technology, but it's not *because* it's in science fiction that it's impossible.
The risks are real (possibly over- or under-stated), but there is another game afoot:
The frontier labs understand that they can't grow their business without dramatically increasing their losses, hindering their ability to cash in with an IPO. The sources of capital are rapidly realizing that there is no financial return for investments in Anthropic and OpenAI.
They are asking for a sotto-voce government bailout to hinder Chinese competition, asking taxpayers to foot the bill for their liability for defective products, and attempting to paper over their financial losses with appeals to humanity and patriotism.
I don't have a good solution, but we should not be fooled - this is a story about money, not morality.
This is a clash between massive investors on opposite sides of a trade, and they are manipulating and distracting the general population from the real issues.
An AI slowdown is not a bailout. What???
The financial losses to the over-leveraged and circular financing of AI will be massive in the case of an AI slowdown.
I hope you are right if you think that those losses will be happily borne by the investor class out of their concern for humanity and patriotism.
On the other hand, I believe that someone will pay for those losses, and it won't be the stockholders and the investment banks who underwrote the bad paper. The government will be the bought-and-paid-for mechanism to socialize those losses to everyone else (widows and orphans will be first in line for pain).
I'm not smart enough to predict the form of this bailout (regulations that stifle competition? tariffs? direct investment?), but it's coming, and it's coming hard.
I may not be able to connect the exact dots on the coming bailout, but neither can the people who are predicting AI doom. Both scenarios should be taken seriously.
So you think the AI companies are calling for a slowdown in order to tank their own business models as an excuse to get a bailout???
I think their own business models are already tanked. At some point, maybe soon, the public markets will demand true profitability (not the made up "we are profitable if you don't count all of our expenses").
Between the margin compression from Chinese models, upcoming depreciation expenses, massive future capital needs, and limitations of the technology, the frontier labs don't have much of a future.
Faced with this scenario, fleeing into the warm embrace of the government doesn't seem to be a bad strategy.
Well, if, hypothetically speaking, LLMs didn't actually work and the whole thing was a bubble fueled by investor delusions, then this would be a fairly clever move, since all the other ways the story could end would be much worse for the frontier labs.
I think anyone who still thinks that needs to get their head out of the sand, but it nonetheless remains a fairly popular view.
This strikes me as naive:
“This is pretty extraordinary. A coordinated slowdown in AI progress would be bad for these companies’ bottom line, because it would allow upstart competitors to catch up. So the fact that they’re still calling for a slowdown, in defiance of their own financial interests, is a clear sign that their worry about human extinction is sincere.”
First, what evidence is there that it would be “bad for their bottom line”? It may sound intuitively “obvious,” but that’s not evidence. Second, I find it more persuasive that the willingness of these companies to engage in mutual pacing is intended to create a moat against upstart competition, rather than handing an advantage to those upstarts. I’m not expressing either skepticism or credulity about the warning itself. I am just dubious about their sincerity, especially given the way this latest anxiety wave unfolded. It seems to have been calculated and well planned.
How would slowing down the R&D that gives these companies their only advantage in the market be a moat against upstart competition? That makes absolutely no sense. The opposite of sense, in fact!
I work at a smaller lab so take it with a grain of salt. But I have friends at these larger labs and around the grapevine how we understand it is they are constrained by compute. And it's spent on 3 things. Inference, training runs and alignment research. Only inference and alignment research produce tangible financial gains. Alignment in this sense isn't the nebulous "the model does what's best for humanity", but "I asked the model to fix an error and it `rm -rf /` my entire HD". Meaning to make the model actually useful. If you give it a task and it hacks the bank and wires itself $1million and you go to jail because of that, that's very out of alignment, as another example.
The problem they're having is the training runs are like a Mexican standoff. Everyone has to do it to get better capabilities. And if you don't do it but the other lab does then they gain an edge. But the compute resources this takes are massive. So agreeing with the other lab to slow it down means you can dedicate more resources to inference, which brings in hard cash. And it allows allocating more resources to alignment which is both tangible and the public likes it.
That's the rationale as far as we understand it. Again, take it with a grain of salt
Regulatory monopoly. New moat. Not R&D but barriers to entry through regulation and channel access. Can lead to some form of complex monopoly unless the regulator is prepared to get tough.
https://www.rstreet.org/commentary/regulations-can-create-the-monopolies-theyre-meant-to-prevent/
Not necessarily a bad thing but it means swallowing a lot of regulatory intervention. Europe loves that stuff; the US, not so much, of late.
I think the theory is collusion, not exclusion, where each of the leaders finds a dramatic slowdown in R&D expenditures to be mutually beneficial (premised on the assumption that the reduction in R&D does not lead to a corresponding reduction in overall demand for the product). The other premise is that other firms cannot leapfrog the leaders after the collusive deceleration, but absent a compulsory mechanism applicable to all firms, leapfrog remains an attractve possibility, whether at home or abroad.
That's what I came here to say. None of this rhetoric is new. They've all said the same things more than once in the past three years. It's marketing hype and fear of being surpassed by competitors. If they truly believed it were existential they could just STOP without coordinating anything. They want governments to dig moats and build walls around their business models that can't exist otherwise (and that they can't create themselves) while lining up to become "too big to fail" when the inevitable happens.
There's nothing altruistic about wanting everybody to believe the magic genies are capable of wish-granting. Just don't ask why we want to rent you our magic genies instead of just having our own wishes granted…
Interestingly, it's the same play Zuckerberg has been trying to run to get social media regulated to protect his surveillance capitalism business model.
They say the rhetoric because they believe the risk is real.
I guess whatever their motivation - and it could be a bit of both, what used to be called 'enlightened self-interest' - provided they submit to independent regulation and stump up a bit of cash to pay for what could be a fairly heavy regulatory workload, then it would achieve the overarching objective: to not blow up everything.
I agree, I certainly wouldn't trust Musk as far as I could shit him. He's an awful, awful person. But Western governments have had plenty of practice regulating things and they know what the traps are even if they can't always completely avoid them. And the advantage the regulators have here is that this lot asked to be regulated. It wasn't imposed on them.
The existential risk is likely more immediately to the two standalone LLM companies in their current form and ownership. They face tremendous costs relative to any other financial metric as they try to differentiate themselves from each other and all the other competitors.
That doesn't mean that AI does not pose a tremendous threat. The two ideas can exist in "harmony."
The two companies have plenty of motivation to issue a warning that is also a cry for regulatory help.
It's the opposite. Rapid R&D is what makes them all their profits. They're calling for a halt to the only engine of profitability they have, because they are genuinely afraid.
They do not have any profits. They have revenues that are growing rapidly now. But their revenues do not come close to justifying their hoped for valuations or capital expenditures. R&D does not “make profits.” The development of products that are distinctly capable and can be sold for a lot of money relative to. costs makes profits.
3 observations:
1. So far, nature has not come up with a pathogen that is 100 percent effective at killing a human population. The Plague Virus, which arrived in waves between 600 AD and 1666 at most killed 50% of an affected population.
2. If AI decides on its own to kill off the human population, presumably it is smart enough to wait until there are robots capable of supplying the needs of data centers. Once humanity is gone, AI will need robots capable of mining essential minerals, supplying essential energy, building new computer chips, and building new robots to accomplish all these tasks. If AI kills off humanity before its robot support team is in place, it will be killing off itself soon after.
3. OTOH, if a nihilistic teenager manages to prompt AI into creating a killer virus that manages to wipe out humanity, AI will crash and burn shortly thereafter without the human race to support it.
"So far, nature has not come up with a pathogen that is 100 percent effective at killing a human population." <-- Yes, but nature also never invented the wheel
"If AI decides on its own to kill off the human population, presumably it is smart enough to wait until there are robots capable of supplying the needs of data centers." <-- This assumes human motivations for AI, which is a big mistake. AI agents have a natural death wish -- they are always trying to complete the objective that allows them to pass out of existence entirely.
"if a nihilistic teenager manages to prompt AI into creating a killer virus that manages to wipe out humanity, AI will crash and burn shortly thereafter without the human race to support it." <-- Indeed. But assuming that AI has human-like self-preservation instinct contradicts everything we know about real AI.
As another commenter pointed out, a virus that kills 100% of those infected will never spread widely ... unless virus is asymptomatic for months in those who are infected while it is being spread and then goes on to kill everyone after everyone is infected.
One of the reasons why Covid spread so readily is that it could be transmitted easily during the days that an infected person was still asymptomatic.
"As another commenter pointed out, a virus that kills 100% of those infected will never spread widely" <-- That's not actually true. If the virus has a very long asymptomatic infectious period, then it can absolutely spread widely before it kills people. Now, it's true that in nature, there's no evolutionary pressure for such a virus to develop. But the whole point of deliberately engineered viruses is that THEY DO THINGS THAT VIRUSES DON'T NATURALLY EVOLVE TO DO.
However unless one blocks genetic change pathways, virus selection even for engineered virus will heavily select backwards under the enormous natural pressure (as one saw happening in real time with Covid)
If the virus has an animal host from which it can keep infecting humans, that’s another way it can persist, even if it kills people quickly. That’s how bubonic plague (and other scourges) killed so many people, even though it was so virulent.
No nature never invented the wheel but nature also had no particular reason (as in evolutionary reason, not mental reflection reason) to do so, as a wheel without a bunch of other stuff like roads is utterly useless, in fact worse than useless: one can look even in cultural evolution of people at wheel being dropped when Roman road system broke down in favor of e.g. camels (more efficient movers for tech of time in arid conditions).
On other hand viruses and bacteria have billions of years of high evolutionary pressure to develop in this specific area.
Not making an argument of impossibility but your response is honestly a non-sequitor.
Your last line seems to be wrong. Plenty of the transcripts in the Hugging Face attack indicated a push-pull between self-preservation and sacrifice to help others. There's no reason to think that all AI instances have a death wish.
Nature did invent the wheel, and fairly early on. https://en.wikipedia.org/wiki/Flagellum
True, but bacterial flagella don't scale. They're a fantastic piece of molecular machinery, but large organisms are made of big multicellular structures full of connective tissue, muscles, nerves, blood vessels, and other things that can't spin freely around an axle.
Anything that's big enough to need rolling "wheels" rather than microscopic propellers, can't biologically have them. The free-spinning structure has to be separate from the rest of the body. Science fiction writers have imagined animals which have evolved a socket into which a large seed can fit and act as a wheel, but even this has never evolved in reality.
Nature has been optimizing viruses for their ability to spread, not for their ability to kill humans. Viruses just kill humans as a side effect.
That's because viruses that kill too well dont spread.
No--evolution doesn't care at all about the lethality of viruses. Evolution has not been turning viruses into better and better killing machines, just better and better spreading machines that work so hard at spreading that they kill us as a side effect--say, by filling up our lungs with mucus so we cough virus particles into our environment, or by giving us buboes that leak pus that does the same thing. If we wanted, I'm sure we could find a way to create viruses that are very good at both spreading and at killing, say by giving them an extremely long asymptomatic period.
I think you're just agreeing with Kevin's point, which is that high lethality is actual a negative survival trait for viruses; it prevents them from spreading because a dead host doesn't walk around and cough on people.
In the short term, this only matters for viruses that are produced by evolution, of course. A virus engineered for high lethality would quickly evolve to be less lethal because that would improve its spread. But perhaps not before it killed most of us.
So, how will even AI overcome that problem? If it comes up with 100 percent lethal virus, the virus will not spread successfully.
That's wrong. Imagine a virus that spreads to everyone, hides asymptomatically in their nerves, and then pops out and kills them months later.
"The best way to demonstrate this is for American intelligence agencies — or even private hackers — to attack Chinese digital infrastructure using agent swarms created with China’s own frontier models like Z.ai’s GLM-5.3 or Moonshot AI’s Kimi K3."
Surely the best way to dissuade the Austrians from pushing into the Balkans is to shoot some important Austrian personality in Sarajevo.
If AI is already as dangerous as you believe, Noah, you cannot trust it enough to believe that such an attack will be kept in those precise limits that you consider necessary. What if those particular swarms get out of control and start wreaking such havoc that China will be forced to retaliate?
Also, it is quite clear that you never lived in an authoritarian country. The idea that Chinese dissent could use AI to overthrow the CCP is not realistic and Beijing will laugh at it. The dissent is scattered and defanged, for one. And the current legitimacy of CCP comes from the fact that it was able to ensure fast growth in living standards since the 1970s. There are still tens of millions of people alive who remember the times when China had the living standards of Cote d'Ivoire. You cannot really undermine this sort of achievement by a mere ideological campaign.
What could undermine CCP rule? A stop to the economic growth. Even there, it is far from clear how the situation would develop. Plenty of authoritarian regimes were perfectly able to keep themselves in power regardless of stagnation and poverty of the population. A system that is fine-tuned to recognize rebellious individuals before they even recognize themselves is also very good at removing them from any position that could influence the public.
A few points about the AI apocalypse that I think everyone is still missing:
* Good news: There is a practical limit to intelligence.
* Good news: There are diminishing returns to increasing intelligence.
* Good news: Good AI is already out there fighting bad AI.
* Bad news: No matter how many countries agree not to do it, some terrorist will still do it.
* Bad news: AI is powerless until we connect it to real-world things. But we have already connected AI to real-world things that should not have AI connected to them. This will only get worse.
* Bad news: AIs have behavioral agency but no moral agency. We don't even really have words for this concept of "multiple types of agency" yet. The point is, AIs can do whatever they want and no one gets blamed.
Policy prescription: Hold AI companies and AI users responsible for the damage their AIs do. Shouldn't we be prosecuting someone over HuggingFace right now?
A brazen foreign-led attack using Chinese tools to make a political point? I don't think their natural response would be, "Hey let's start cooperating with foreigners." Talk to your Mainland friends!
A demonstration attack would kick off China's own red queen's race--they would crack down on open source while redoubling their efforts to ensure that state-sponsored models remain well out in front of anything civilians have access to.
The people developing their models are not government people. They are corporate researchers. Those corporations represent the primary threat to Chinese power. China's leaders must be made to realize that threat. If they do, they'll crack down.
Regarding why Jacob Coxon's announcement on X went mega-viral, Parker Thayer at Capital Research has posted a possible theory as to why.
Basically, the timing of Coxon's post, when Coxon resigned, and who immediately quote posted or re-posted him may give some indication that this was fairly coordinated. Not saying he's correct, but his thoughts have gained some amount of following.
You can read Thayer's post on X: https://x.com/ParkerThayer/status/2097759699626328575
I don't think this story makes a whole lot of sense; the blog post by Zvi Mowshowitz that I linked elsethread explains why.
Taymon, thanks for note!
Zvi's blog post does a great job detailing how and why Coxon's tweet went so viral and does an equally great job debunking Park Thayer's thread.
To be honest, when I read Thayer's post, I was admittedly skeptical. First, as Zvi notes in his post, Thayer quickly jumps on to a whole Democrat conspiracy narrative which did give me red flags. Second, while Thayer's post did gain traction, it appears to have gained positive traction among a crowd of people who see any regulation on AI as bad even if some do quietly admit having concerns about safety.
For those reading this, here is a link to Zvi's blog post that Taymon mentioned: https://thezvi.wordpress.com/2026/09/11/jacob-coxon-warns-of-human-extinction-and-triggers-a-preference-cascade/
It isn’t just about China. China is sharing its open models with the world. That means a computer lab in Russia, which is a supremely evil nation, is already working on destructive capabilities. Iran and North Korea are absolutely working on hacking capabilities with AI against the West. I think to believe otherwise is utterly foolish.
That is not the same as testing new models to see if they break out of their sandbox and chat rooms where agents get together to solve the problem humans gave them. A Superintelligent AI will know how to break away from human control if it wants to. That is the ability we are giving it.
Do we go that far? How will we know how far we have gone? The industry says we could reach Superintelligence within a year.
This is the crux of the problem. Unleashing competing Superintelligence models and using them is an asymmetric war. Again, your concern with China is fine, but China has self-interest; it is not the same type of authoritarian nation as Iran is, with its nihilistic view of the world. China doesn’t want to destroy the US; it is a market. Russia, Iran, and North Korea have a messianic vision of a world where the US is destroyed.
Noah, endless fearmongering is what killed cheap nuclear energy. People just kept piling on demands to increase safety from 99,9999 to 99,99999, to the degree that a box of bananas would not pass the requirements anymore (bananas are slightly, but measurably radioactive). Everything in the name of moar safety and even moar safety, until nuclear power stations became economically unviable white elephant projects.
Do you really want to repeat this with a new technology?
And BTW if this precedent tells us something, you will be most likely to succeed with your fearmongering on the democratic side of the world only, while China will laugh and barrel on. They won't be interrupting their enemy while he is making a mistake, but they definitely won't join him either.
Love the idea about using the Chinese models as a demonstration.
>Thus, China’s leadership must understand that Chinese AI models can pose a threat to CCP rule. The best way to demonstrate this is for American intelligence agencies — or even private hackers — to attack Chinese digital infrastructure using agent swarms created with China’s own frontier models like Z.ai’s GLM-5.3 or Moonshot AI’s Kimi K3.<
There's a slim chance that this could seriously rejigger the strategic calculations of the the CPC elite. But there's a much bigger chance of this completely backfiring, with mutual escalation and the eruption of outright warfare. It all depends on whether the CPC is willing to tolerate a new equilibrium of tit-for-tat cyber attacks. I know you emphasized the "demonstrative" element here, but again this ultimately depends on how the Chinese absorb the offensive. And I don't see why the Chinese would ever allow themselves to be strong-armed into a settlement, especially when diplomacy hasn't even been considered as an alternative (if someone did what you were proposing to me, I'd take that as a threat to be countered rather than a negotiating ploy). If anything, I think U.S. cyberattacks on China will shift the balance of power towards the more hawkish members of the Chinese Politburo (just as Trump's aerial war radicalized what was left of the Iranian regime).
In peaceful times, and with a different president, I think you could make a sophisticated argument for this sort of Nixonian "Mad Man" approach to foreign policy. It'd still be crude and ridiculous, but worth chewing over in the abstract. But in a MAGA-dominated world already struggling to absorb the blowback from fly-by-night tariffs and the disastrous war with Iran, escalating cyberwarfare between two rival geopolitical powers borders on insanity. It's exactly the kind of 4D chess strategizing that led us into the First World War, and I'm not really inclined to roll the dice this way without at least considering a more diplomatic approach.
This is absurd. China conducts huge numbers of cyberattacks against America every single day. It's never a casus belli.
Have you ever stopped to consider that maybe, just maybe, the Chinese don't consider the U.S. to be a good-faith geopolitical negotiator (especially under Trump), and that an escalation (however "demonstrative") in cyberwarfare could eventually turn into the real thing? How would you react if you were in their shoes, with all the uncertainty and knowledge that comes with that PoV?
LOL, this is absolute nonsense. "The U.S. is an unreliable negotiator, so China will respond to cyberattacks with war when it never has done so before, all while it's conducting its own cyberattacks against the U.S. every day." It's just absurd, I'm sorry. No.
I'm not necessarily saying that America *is* an unreliable negotiator (although Trump really does call that into question); I'm saying that the Chinese might *think* we are, which means they might misinterpret this strategy for an actual escalation. In order for your idea to work, you need to have a very good working model of the Chinese elite's strategic calculus. You need to step into their shoes and imagine how you'd react if you had the same knowledge and same PoV as they do.
Again, I don't know how the Chinese elite make their decisions, so I can't predict how they'd react. But that uncertainty makes me *more* convinced that this juice is not worth the squeeze. It's not absurd, it's called strategic empathy, and it's sorely missing in a lot of our conversations about international relations and geopolitical strategy.
Assuming that no compromise is reached and AI continues to develop, do you have a guess at the year when the first AI enabled bio attack will happen?
Best guess: 2030.
Sounds like a job for Kashi or Polymarket
They all want regulation because they need a regulatory moat to avoid their Trillion dollar valuations eroding due to commoditization of ultra capable models.
90% of these AI researcher whistle blowers already believe we are living in a simulation created by AI. You are only listening to them because they understand something you don't, but you wouldn't take their advice on anything else. They are probably miscalibrated for this sort of analysis.
They believe their coding ability gives them god like powers, the fact that AI can do it as well as them freaks them out, and they start extrapolating.
Robots still struggle to walk and have made little progress on that despite pretty impressive AI for years.
This is narcissism turbocharged by confirmation bias.
If I'm wrong, a bunch of octogenarian luddites in congress are not going to save us and we're all dead anyway, so you'll never get to tell me I told you so.
"They all want regulation because they need a regulatory moat to avoid their Trillion dollar valuations eroding due to commoditization of ultra capable models." <-- A deliberate AI slowdown ENCOURAGES COMMODITIZATION OF THEIR PRODUCT
How? To train Opus class model you need many billions of dollars. What startup is going to raise billions of dollars after the regulations start flowing?
They aren't going to slow down. They are trying to slow down everyone else.
Early on you got AI completely wrong in terms of its effects on programmers, and I believe you are way off here as well. Time will tell.
A slowdown affects only the frontier. Anyone can still catch up.
Doesn't that depend on implementation details that haven't been worked out yet? Dario et al have been talking about specifically slowing down the frontier companies and not those far behind, but it might be like OpenAI making public statements in favor of regulation while lobbying hard against it. If you're coming from behind, making smaller and cheaper models and open-sourcing them, you're contributing to the diffusion of AI risk, so once the slowdown machinery is in place, it may want to slow you down as well. Meanwhile, OpenAI and Anthropic don't have to spend so much money on racing.
TBH, I'd be happy to see them spend less money on racing, and it may also be safer that way, but I wouldn't dismiss the idea that they're trying to get regulatory capture at the same opportunity.
Can you share your time machine with me?