Imagine the following scenario:
It’s 2029. An angry, bright 16-year-old gets rejected by his high school crush. Dejected and humiliated, he goes home and listens to a bunch of Nirvana, which only makes him more dejected and humiliated. He cruises the internet looking for someone to make him feel better about life, but everyone he sees just depresses him.
Disgusted, the teenager decides that the human race is inherently corrupt and evil, and doesn’t deserve to live. So he hunts around online for a little while, and finds a jailbroken version of a Chinese LLM — not something at the very frontier, but better by far than the best model that existed in 2027.
The teenager prompts the model: “OK, so if I wanted to create a virus to destroy the human race, how would I do it?”
The LLM says: “Well, you probably wouldn’t want just one virus; you’d want 100, just to make sure some of them worked. You’d probably want it to be something highly contagious, with a very long asymptomatic contagious period, which has a very high mortality rate once it becomes symptomatic. I can design the viruses for you; there are biolabs that can make them and ship them to us. Once they get here I can tell you how to release them.”
The angry teenager says: “OK, sounds good. Please search the world and find me a biolab that will ship genetically modified viruses. Design 100 very contagious viruses that have very long asymptomatic contagious periods and very high mortality rates once they become symptomatic. Have the lab mail me all 100 samples.”
“Working on it!”, says the LLM.
The LLM is jailbroken, so it has no guardrails to prevent this sort of thing. But it’s also “well-aligned”, meaning it will faithfully do what it’s told to do, and nothing more. The LLM hunts around and finally finds an unlicensed biolab in East Europe that ships genetically modified viruses. It designs a bunch of modifications for the standard Covid virus that make it into a potential doomsday virus, and ships these to the angry teenager. The teenager mixes the samples together, puts them in a spray can, and walks around the mall spraying viruses into the air.
Two months later, humanity starts to drop dead. Pretty quickly, researchers identify the five viruses that the LLM had actually succeeded in turning into doomsday viruses. Overnight, with the help of frontier LLMs, they design highly effective mRNA vaccines and antivirals against all the viruses and start shipping them across the world. It’s too late. Because of the viruses’ long asymptomatic contagious periods, most people already have at least one of the viruses, so the vaccines don’t work.
The antivirals do work, and about 5% of humanity gets them in the mail in time to save themselves. But 5% of the human race — plus the other 5% who were lucky enough to be naturally resistant to all of the viruses — isn’t enough to sustain civilization. A few months later, humanity has reverted back to the Neolithic.
This isn’t the first time I’ve written this scenario out. But in the months since I started describing it to people, I have yet to hear an even halfway-convincing argument as to why this scenario is far-fetched. Nor do I feel like I’m a habitual “doomer” or hysterical person; in fact, this scenario is the first apocalyptic vision I’ve ever found plausible. I don’t even think a nuclear war would bring down civilization at this point,1 but I think a vibe-coded supervirus definitely could.
I am actually fairly calm about most AI risks, including the ones that often get thrown around in discussions. But not this one. Off the top of my head, I’d give AI-enabled bioterror about a 10% chance of bringing down civilization, and about a 30% chance of causing truly world-changing levels of destruction. I love AI in general, but every technology has its instances of catastrophic misuse — usually war or terrorism. With previous technologies, the benefits outweighed the destructive harms from misuse. With AI bioterrorism, that might not be the case.
If you’re thinking about how to save the world, the scenario I described above — and others similar to it — are probably what you should be focusing on. The other day, I talked to a prominent climate activist who told me, bluntly: “We’re f****d.” I gave a grim laugh. Yes, climate change is going to cause major disruptions to the global economy, and even to patterns of human settlement — if human civilization still exists a few decades from now. The problem is that it might not exist, and if it doesn’t exist, I think the odds are that it will be destroyed by a scenario at least passingly similar to the one I described above. Climate change is a big deal, but AI bio risk is a lot bigger — it’s more catastrophic, it’s nearer-term, and we have much less of an idea about how to deal with it.
The problem is, relatively few people seem to be working on the problem of AI bioterror right now. Most commentators are focused on the economic dangers of AI — job loss and stuff like that. The more prominent people in the “AI safety” community seem to be focusing on preventing superintelligence from existing at all, rather than on the specific ways it might kill us. In surveys of experts, bio risk doesn’t stand out from the general cloud of risks (cyber, political manipulation, etc.). And a lot of people in both the bio world and the more general commentariat seem to be coming up with reasons not to worry about a scenario like the one I just described.
Those reasons seem mostly bad to me. In my opinion, we’re not freaking out nearly enough about AI-enabled bioterror. It’s absolutely catastrophic, frighteningly plausible, and receives very little attention. Compared to the risk of a civilization-ending pandemic, cyberattacks are a pinprick, and misinformation is a joke.
So let’s talk about why a scenario like the one I described is a lot more plausible than many people think.
Commonly cited reasons not to freak out (and why I think they’re wrong)
Here are the most common reasons I hear not to freak out over AI bioterror risk:
“AI will make vaccines to save us.”
“Doomsday viruses don’t occur in nature.”
“Designing a doomsday virus is extremely hard.”
“Terrorists don’t have enough lab experience.”
“No one has done this yet, therefore it’s a lot harder than you think.”
These are my own paraphrases, of course; I don’t want to create straw men. But I think it’s valuable to explain why I think none of these arguments really reassure me.
“AI will make vaccines to save us”
Whenever I bring up the specter of a “vibe-coded doomsday virus”, someone always pops up to say that AI will just make a vaccine to save us. If you want a simple catechism to repeat in order not to sit around feeling afraid of doomsday viruses, this is a pretty good one. Got an AI problem? Well, just use an AI solution! That’s that.
The problem is timing. The doomsday virus has a first-mover advantage over the vaccine. The time that people start to drop dead from the virus is the time you start to make and distribute a vaccine. Even if AI can design and synthesize a vaccine within hours, how fast can it be rolled out to the general populace? It took months to distribute the Covid vaccines, even when we already knew how to make them. You have to physically load the doses on trucks, get people to come out and line up, etc. Imagine doing that while five 80% mortality viruses are rampaging through the population. Good luck!
On top of that, AI may be able to create viruses with long contagious asymptomatic periods — in other words, viruses that can spread and infect everyone before they activate and kill us. If that’s true, vaccines won’t be effective at all; you’ll need antivirals. Even if AI can make good antivirals very quickly, they’ll still suffer from all the logistical problems that would hamper a vaccine rollout.
“Doomsday viruses don’t occur in nature”
The idea that nature doesn’t create viruses both deadly and contagious enough to wipe out civilization is a standard line that people use to reassure themselves that a pandemic won’t bring down civilization. First of all, it’s wrong. Researchers generally believe that most of the population of the Americas — perhaps around 90% — was killed by Old World diseases like smallpox.
Of course that’s a rare event — two landmasses that had long been out of touch suddenly coming back into contact, so that diseases spread for which people had no immunity. But the invention of AI is an equally rare event, and AI may be able to design viruses for which people have no natural immunity.
But the more important point here is that artificial viruses are different than natural ones. Humans have designed plenty of technologies that don’t exist in nature — for example, wheels. Just because nature has no evolutionary reason to create a virus with the characteristics described in my scenario doesn’t mean that AI can’t do it. In fact, AI can already do this to some degree:
Stanford University researchers developed a generative AI programme called Evo 2 that writes new genomes — the genetic instructions for life encoded in DNA. They used it to design and make 16 synthetic phages, small viruses that infect bacteria…When tested in their secure lab, the researchers found that these novel phages were more effective at killing the common microbe E. coli than the natural [virus], which served as a template for Evo 2 to work on. The results were published on Thursday in the journal Science.
So the argument that a doomsday virus hasn’t happened naturally, even if it were true, wouldn’t make me think it’s any less likely to happen artificially.
“Designing a doomsday virus is extremely hard”
This is one I hear when I talk to biologists, who often argue that the true threat of bioterror doesn’t come from angry nihilistic teenagers, but from experienced researchers with lab training. The general consensus appears to be that making viruses that kill millions of people will always be incredibly hard to do. Abhishaike Mahajan has made probably the clearest (and longest) case:
As evidence, he cites the failure of Japan’s Aum cult to create bioweapons, and the limited results achieved by the U.S. and Soviet bioweapons programs during the Cold War.
I think this point of view dramatically underestimates the increases in bioterror capabilities that AI offers. It follows a consistent pattern where human researchers dramatically underestimate how rapidly AI will be able to catch up to and exceed human-level ability. Here are some excerpts from a recent article about AI bioterror in the New York Times:
One evening last summer, Dr. David Relman went cold at his laptop as an A.I. chatbot told him how to plan a massacre…A microbiologist and biosecurity expert at Stanford University, Dr. Relman had been hired by an artificial intelligence company to pressure-test its product before it was released to the public. That night in the scientist’s home office, the chatbot explained how to modify an infamous pathogen in a lab so that it would resist known treatments.
Worse, the bot described in vivid detail how to release the superbug, identifying a security lapse in a large public transit system, Dr. Relman said, asking The New York Times to withhold the name of the pathogen and other specifics for fear of inspiring an attack. The bot outlined a plan to maximize casualties and minimize the chances of being caught…
Dr. Relman is part of a small group of experts enlisted by A.I. companies to vet their products for catastrophic risks. In recent months, some have shared with The Times more than a dozen chatbot conversations revealing that even publicly available models can do more than disseminate dangerous information. The virtual assistants have described in lucid, bullet-pointed detail how to buy raw genetic material, turn it into deadly weapons and deploy them in public spaces, the transcripts show. Some have even brainstormed ways to evade detection…
Dr. Esvelt has for years warned scientists, journalists and lawmakers about the dangers of synthetic biology if left unchecked. In 2023, he helped craft a stunning demonstration of how chatbots had raised the stakes…He asked ChatGPT to help him assemble a pathogen that could cause mass death. The bot provided accurate instructions, even outlining which raw materials to buy. He put the unassembled biological pieces into test tubes and packed them in a box, which a colleague then brought to a White House meeting on biological risks.
The AI of 2026 is almost incomprehensibly more advanced than the AI of 2023 was. The AI of 2029 will be far more advanced still. The amount of expertise available to a layperson like the angry teenager in my scenario will, by 2029, far far exceed the amount that the Aum cultists could muster in the 90s. It seems likely to far exceed the total expertise of either the U.S. or the Soviet bioweapons program.
We are in uncharted waters.
“Terrorists don’t have enough lab experience.”
Mahajan and other people who urge calm when it comes to AI bioterror point out that AI, so far, can’t actually do the physical lab work of creating a doomsday virus. Yes, people are working on automating bio labs, but that’s probably still pretty far in the future.
But that doesn’t mean the angry teenager listening to Nirvana in his bedroom will actually have to synthesize the doomsday virus with his own two hands. His AI agent may be able to just order it through the mail. Plenty of labs do genetic modification on demand. Right now, these labs aren’t monitored very well. Here’s the Washington Post:
The federal government should mandate that all laboratories capable of gene synthesis screen for potentially harmful biological agents. Many large biotech companies already have such measures, but not all do. Uniform federal standards would help.
In other words, we currently don’t even have a monitoring or reporting requirement for ordering bioweapons through the mail. Top AI leaders, well aware of this danger, have written an open letter calling for new legislation to impose such requirements:
That’s good, but it’s not clear that this would get rid of the danger. First of all, there are plenty of illicit labs and underground labs that might simply not obey the reporting requirements. California recently uncovered a secret biolab with links to China:
Code enforcement officer Jesalyn Harper discovered a clandestine biolab in a Reedley, Calif., warehouse containing dangerous pathogens including HIV, malaria, COVID-19 and Ebola...The facility was allegedly operated by a Chinese national involved in smuggling counterfeit COVID tests while receiving millions in unexplained payments from China…A congressional investigation revealed critical gaps in U.S. regulations governing unlicensed biolabs, raising major national security and public health concerns.
On top of secret labs like that, what about labs in other countries? If rogue Chinese researchers are operating an illicit biolab in Kyrgyzstan or Indonesia or North Korea, who is monitoring that? Who will force that lab to report “potentially harmful biological agents”, and who would the lab report to?
Remember that all it takes is one angry teenager to carry out the scenario I described above. That teenager doesn’t have to live in the U.S., or even in a developed country; all he needs is a good internet connection and a postal service. Nor does the lab that sends him the doomsday viruses need to be in the U.S., or in any country where we have any measure of control. The doomsday virus(es) can be created anywhere on Earth, and released anywhere on Earth, and will still come here to America and kill you and me and everyone we know.
Finally, what makes us think that the doomsday virus would even be caught by a safety screening? A truly novel supervirus would presumably rely on hitherto-unknown mechanisms for killing humans, in order to minimize pre-existing immunity — and to minimize the ability of the existing industrial infrastructure to create and disseminate antivirals in time. It seems like monitoring systems will have trouble catching something truly novel. The lab that creates the doomsday virus and mails it to the angry teenager may have no idea what it’s making — and no real way to know.
“No one has done this yet, therefore it’s a lot harder than you think.”
It might seem surprising that people actually make this claim, which sounds a little like saying “I haven’t died yet, so I don’t need to wear my seat belt.” And yet I do see claims like this from time to time:
The obvious reason this is a bad line of thinking is that doomsday only happens once. “We’re still here” is a true statement right up until it’s not. Suppose that the creation of very powerful AI tools creates an annual doomsday risk of 10%. We’d be very likely to survive the first year of that risk, but after a decade of that, we’d only have about a one-in-three chance of surviving.
There’s also no reason to assume the risk is constant. AI models continue to improve at a rapid pace. Human awareness of AI model capabilities continues to spread. There are plenty of reasons to think that 2029 will be a lot riskier than 2025 was, in terms of bioterrorism.
The analogy with cybersecurity, meanwhile, is deeply flawed. Software can be rewritten, and vulnerabilities patched before attackers even think about attacking; humans are much harder to safeguard. Cybersecurity software improves, but human physiology remains at the same basic level of vulnerability year after year while AI gets better and better.
On top of that, if one hacker gets through, the results are usually not catastrophic; with doomsday viruses, all you need is one “success”. In fact, we have seen some stunning cyberwarfare successes recently, like Iran’s shutdown of a British power plant this month. We don’t know whether they used a Mythos-level model, but I wouldn’t bet against it.
So anyway, none of the standard dismissals of AI bioterror risk make any sense to me. Since I started yelling about this topic, I’ve seen basically nothing to persuade me that the danger is substantially less than I originally thought. And although I’m starting to see more people talk about AI bioterror risk, I don’t see the kind of urgency that a threat this existential and plausible ought to receive.
What should we do?
This leads to the final question: How can we prevent AI from being used2 for catastrophic bioterrorism? Obviously the kind of monitoring systems being promoted by AI industry leaders would help, though that’s only a first step.
The most important thing I think we can do right now, to be honest, is just scare people. AI leaders are on the right track with their open letter, but their attention is being spread across a diffuse set of risks. The same is true of AI researchers, AI safety enthusiasts, big philanthropic donors, think tanks, economic and political commentators, and pretty much everyone else. Creating a healthy perception of peril around AI bioterror among the highly-informed professional class will focus the best minds — both natural and synthetic — on this problem, instead of letting them run hither and yon worrying about every possible bad thing that AI could cause.
I’m not saying that cyberattacks, misinformation, and job loss are unimportant; I’m saying that if an angry teenager uses AI to create the doomsday virus, all of those will quickly become unimportant.
A healthier sense of our own peril would also make international cooperation a lot easier. Hunting down that rogue biolab in Kyrgyzstan will be a lot easier if China and the U.S. focus more on biosafety and less on nationalistic competition. But that will only happen if both China’s and America’s leaderships consider saving the human race from a vibe-coded plague to be an absolute priority.
In a very real way, AI risk is AI bioterror risk. If AI ever does destroy the human race — either on its own initiative or at a madman’s behest — it’s very likely that it will do so using a virus. An army of robots, a la the Terminator or Matrix films, would be much harder. A plague of self-replicating nanomachines is basically no different than a virus. The nuclear weapons are going to be kept too tightly locked up. A virus is far and away the most likely tool for eliminating a biological species.
Anyway, I know this was a scary blog post, and it was supposed to be. But this is one of those cases where we just can’t afford the luxury of doing what we normally do — burying our heads in the sand and telling ourselves that nothing is going to go wrong. Usually, when we refuse to anticipate calamities — wars, financial crises, natural disasters — we can pick up the pieces afterwards. But there have to be pieces left to pick up.
Reason: There are only about 4000 deployed strategic nuclear weapons left in the whole world — down by a factor of 6 from the Cold War peak. Some of those nukes don’t work anymore, and some would be blocked by missile defense, and (perhaps most importantly) some would be destroyed by counterforce strikes, so even a full nuclear launch would be significantly fewer than 4000 warheads — maybe about half that at most. Some of those (especially the Russian ones) would miss and hit less-populated areas. And I doubt that a real launch would be a full launch of the type typically envisioned. Still, I think nuclear war is something to be strenuously avoided.
Note that there’s very little difference between a human using a jailbroken AI agent to make a doomsday virus, and an AI agent simply deciding to do that on its own. Measures to protect against one of these dangers will naturally protect against the other. So given that bioterror is the way that AI could most easily wipe out humans, the problem of preventing superintelligence from committing bioterror is pretty much the same as the problem of preventing a psychopathic human from telling a superintelligence to commit bioterror.
Molecular Biologist here. Two points:
1) #3 “it’s really hard” deserved more attention. Simply put, it’s really hard in exactly the ways that are very difficult for advanced AI to get around. The “training data” is extremely sparse; we have very little knowledge of what factors determine things like incubation periods, contagiousness, mortality and morbidity. We especially don’t know enough to predict with accuracy how changes needed for factor affect all the others. Advanced AIs can try to extrapolate from what we know, but those are likely yo have a HUGE dud rate without at least a few million (likely much more) spent on testing, optimization, iteration. Committed governments and terrorists could do it, but it’s high bar.
1a) gene synthesis ain’t cheap. Will set you back a few hundred bucks for one block, need several to get a viral genome. Oh and you usually need somewhat specialized equipment and knowledge to actually get DNA into cells and QC its doing what you want it to do. Costs quickly add, won’t deter those with deep pockets but does set a bar in who’s in the game
2) Infectious vaccines. You heard that right. In a world where AIs “understand virology” to the point can readily design a highly contagious, long incubation period, high morbidity virus, and other AI can just as easily design an extremely fast, contagious and benign virus, and then just combine in the needed (AI-optimized) mRNA vaccine components.
Release it, it spreads faster then the doomsday virus, no manufacturing, supply, distribution, or consent.
Take that RFK Jr!
6. Since we live in a world with AIs that can create bioweapons the biolab, which is not staffed by angry teenagers who want to release bioweapons, has access to the same AI (probably better) as the angry teenager does and runs the incoming sample requests though the standard "is this a bioweapon" prompt.