1) #3 “it’s really hard” deserved more attention. Simply put, it’s really hard in exactly the ways that are very difficult for advanced AI to get around. The “training data” is extremely sparse; we have very little knowledge of what factors determine things like incubation periods, contagiousness, mortality and morbidity. We especially don’t know enough to predict with accuracy how changes needed for factor affect all the others. Advanced AIs can try to extrapolate from what we know, but those are likely yo have a HUGE dud rate without at least a few million (likely much more) spent on testing, optimization, iteration. Committed governments and terrorists could do it, but it’s high bar.
1a) gene synthesis ain’t cheap. Will set you back a few hundred bucks for one block, need several to get a viral genome. Oh and you usually need somewhat specialized equipment and knowledge to actually get DNA into cells and QC its doing what you want it to do. Costs quickly add, won’t deter those with deep pockets but does set a bar in who’s in the game
2) Infectious vaccines. You heard that right. In a world where AIs “understand virology” to the point can readily design a highly contagious, long incubation period, high morbidity virus, and other AI can just as easily design an extremely fast, contagious and benign virus, and then just combine in the needed (AI-optimized) mRNA vaccine components.
Release it, it spreads faster then the doomsday virus, no manufacturing, supply, distribution, or consent.
Note that if everyone already has the virus by the time it starts killing people, then even an infectious vaccine won't do much. You'd need an infectious agent that produces large quantities of antiviral in the body.
Also, sure you don't know 100% that a doomsday virus will work, which is why the terrorist releases 100 at once!
Also there are even scarier things that I didn't want to write explicitly about, but some biosecurity people have been telling me about. Happy to talk on the phone sometime if you want!
"You'd need an infectious agent that produces large quantities of antiviral in the body."
Pretty much every vaccine in existence does this! Within a few days of a vaccine (or naive infection), the B-cells that recognize viral proteins ("antigens") divide rapidly and secrete exponentially increasing amounts of IgM antibodies that bind to the viral antigens. These antibodies don't just prevent infection, they can block the spread of virus between cells in the body and target infected cells for clearance. A little bit later on maybe a week or so, T-cell response kicks in too. This produces "antivirals" in two broad flavors: helper T-cells that further stimulate the production and maturation of antibodies, and killer T-cells that specifically seek and destroy infected cells (and can effectively recognize ANY part of the virus, even "hidden" ones that evade the rest of the immune system).
So after about 2-3 weeks, even if this doomsday virus managed to produce absolutely NO adaptive immune response in its "dormant" phase (itself a super-hard problem), a vaccine or "benign" virus would likely kick off a highly effective adaptive response that greatly improves survival when the "morbid" phase activates. And if instead the virus did provoke a even a weak, low-level adaptive response in dormancy, then the vaccine would act like a booster shot, where exponential B- and T-cell activation both occur all over again, but in a much more coordinate and mature way.
"Also, sure you don't know 100% that a doomsday virus will work, which is why the terrorist releases 100 at once!"
Odds are only a few of those spread widely, even if all are highly transmissible. 1) each individual virus spreads separately from the other 99, because you cannot make a massive genome that encodes all 100 variants fit inside a tiny virus, so each very quickly comes under its own selection pressures. Let's say the avg in that 100 has an R0 of 4, and the best is just 20% better, at R0=5. After 10 infection cycles, the most transmissible one spreads 10X farther than the one average ones. This also assumes no cross-immunity between the 100; if there is even to a small degree then the 100 end up competing with each other and you get a few winners.
Vaccine design doesn't suffer from this problem, because what matters are the handful of very short sequences that produce the strongest antibody/T-cell responses. Even pre-LLMs, this could be predicted a-priori from sequence to a fairly decent (now excellent) degree, and can be confirmed very quickly (weeks) by analyzing serum/white blood cells from a few dozen patients. You have plenty of genomic space to daisy chain 100 or more short but highly immunogenic sequences into one backbone, ensuring that all 100 "stay together".
"Also there are even scarier things that I didn't want to write explicitly about, but some biosecurity people have been telling me about. Happy to talk on the phone sometime if you want!"
Thank you, I am sure there are much worse scenarios I can't think of, but I prefer to sleep at night : )
And FWIW, I commend you for thoughtfully calling attention to AI bioweapons as an under-appreciated high level risk. I absolutely share this concern. I just think it's "hard" in a way where more "intelligent" AI doesn't dramatically elevate risk until you get to VERY high levels of both automation and intelligence. But all long-tail risks are "hard", and biosecurity should be higher on people's radars.
I agree with every point that Froberg has made on this thread, but I wanted to chime in to emphasize the evolution point. Once you release this AI designed superkiller, it immediately succumbs to ordinary selection pressures. If the traits that make it super lethal are not adaptive, it will quickly lose them in a few generations. At that point you don’t have a superbug wiping out humanity, you have a tragic but very limited bioterrorism event that kills a few dozen people.
The cleverness of AI really comes into play less than the initial scenario supposes. The selective pressures the AI superbug encounters are emergent phenomena that the AI cannot know in advance because they are largely unknowable.
Isn't "long period of asymptomatic contagion" also quite hard? Contagion requires the virus to replicate a lot of itself, but it's replication and the body's immune response that causes most of the symptoms.
I think instead you'd need a virus that produces mild cold/flu symptoms in its early stage, when it has replicated a lot on the body, and then perhaps goes completely dormant for a few months (somehow) and then comes roaring back in some Ebola-like organ-shredding mode.
Either that or a virus that somehow manages its replication so it reaches a barely-contagious but sub-symptomatic level and then holds there for a long period of time... and it would need to do that in everyone, in spite of wide variations in bodies and their immune responses.
I'm certainly no expert, though. Maybe this is easier than it seems? Are there any examples of naturally-evolved viruses that work this way (even without the organ-shredding secondary mode)?
Yeah, It’s not easier, to my knowledge most human viruses really don’t behave that way.
Incubation/dormancy periods have a fitness cost: energy/complexity needed to evade innate immunity or maintain a latent state when you’re not spreading much. Could have a long asymptomatic shedding period, with active replication but epic immune evasion, but that also takes a ton of energy.
Organ shredding mode also has a fitness cost: shred organs too hard and you wipe out your own host!
Put another way these traits are hard to evolve, much less both engineer and engineer in such a way that they don’t just get quickly mutated out into something more in the virus’ interests.
HIV does not spread quickly at all, compared with COVID, influenza, measles, etc. Airborne, droplet, saliva, fecal transmission all require infection in mucosal tissues, versus deep in the blood, and very fast replication of huge amounts of virus. Fast replication in mucosal tissues causes tissue damage (versus "silently" clearing out a fraction of white blood cells), tissue damage and fast replication of all trigger an innate immune response.
Put it another way, HIV has a highly unusual strategy of directly and persistently infecting immune cells in the blood, which is not easy to combine with rapid contagion.
Creating a Timothy McVeigh-type mass casualty event is no longer easily possible, because public purchase is no longer possible of the ingredients necessary to make a truck bomb.
The FBI needs to be authorized by law to more closely restrict the sale and purchase of any and all materials that could be used to produce bakers dozens of vibecoded killer viruses.
McVeigh used ammonium nitrate and nitromethane, neither of which is substantially regulated. There is a voluntary reporting program so some vendors may report sales they deem suspicious, but that just requires potential bombmakers to find a vendor who won't report them, or structure their purchases as part of activities that make them unsuspicious.
So why don't we have an HIV vaccine that works yet? The natural antiviral agents the human body produces simply aren't able to get the job done in most people who get infected. It took the invention of artificial antiviral drugs before HIV turned from a death sentence into a life sentence.
HIV in the 1980s had nearly all the properties of a doomsday virus anyone could want; the immune system couldn't clear it, and it had an extremely long (mostly) asymptomatic period before its human host inevitably died. The only reason it wasn't an actual doomsday virus is that it could only be transmitted via bodily fluids. If HIV had been as contagious as measles back in the 1980s, we'd all be dead now.
HIV directly infects T-cells, key cells in adaptive immune response. T-cells themselves are an interesting target; silently clearing a portion of them out is a much weaker signal to the rest of the immune system than tissue damage in the lungs or gut. HIV also integrates into the host genome, leading to a large reservoir of "permanently infected cells". And on top of all this, HIV mutates at the very edge of stability, constantly evading the host's immune defense.
It might seem like a doomsday virus, but the key is all of these "odd" traits that make it so challenging for vaccine development/control without antivirals are in fact a coordinated set of traits optimized for long-term blood infection. That's a completely different set of traits than those optimized for high contagiousness and/or rapid spread.
For those viruses, T-cells are a poor choice for preferred cell because they have almost no "contact" with the outside world. Genome integration doesn't matter much if you're doing fast hit-and-run spread, and neither does extremely fast mutation.
Thank you for sharing your expertise on this subject.
I'm the opposite of an expert and don't want to be a pollyanna, but I just get the feeling that this is ascribing superpowers to AI that won't apply to the cussed complications of real world biology.
Which is not to say that the angry teenager couldn't kill 5% of the world population, unfortunately. Reason enough to proceed with great caution.
Taking rabies as an example of a disease that has a long incubation and nearly 100% mortality (luckily not contagious), the vaccine is normally taken post infection. So the conclusion that the vaccine won't work after infection is, at best, only partially true.
It seems to me that viruses constantly and furiously recombine themselves in almost every infected cell, so maybe millions of times in the span of a few hours for every infected living being. That's a massively parallel evolutionary search that has been constantly going on since before mammals walked the earth. It is true that, as you say, there are no particular evolutionary reasons for a virus to be deadly, but it's hard to think that an AI can outdo nature in exploring this space.
I think most of the answer is that smooth terrains are very rare in nature, and wheels quickly become ineffective when medium-big obstacles appear. I do think that if wheel locomotion was a survival imperative (and feasible biologically) nature would have invented it.
That said, I am not completely sure it’s feasible to grow wheels, axels and some kind of engine organically within a single organism. From a mechanical point of view. Maybe it’s possible as a symbiosis of different organisms.
So if your point is that there may be no natural evolutionary path that incrementally leads to a biological wheel locomotion system, while it might still just be possible to design it and assemble it, using engineered biological components, I guess that’s a good point.
Not sure how/if it applies also to viruses but it might. I guess the crux of the problem is that we can now design complex (biological) systems that nature has no reason to invent.
That's not a wheel; it's just a spider that tumbles downhill doing cartwheels.
A wheel, in this context, is an axle connected to a circular object (necessarily completely detached) used for locomotion. The flagellum comes close, but it's a detached "tail," not something round.
I agree that #3 deserves more attention. But I also agree with Noah that at some point someone it going to try this. My prediction is that the first attempt will be flawed and will not wipe out humanity, but will be bad enough that it gets people to implement the safe-guards Noah is recommending.
All humans aren’t equally susceptible to infection or as likely to recover fully.
The Chinese allegedly were mucking around with optimizing viruses for genotypes. Military program. Say….infecting Indians or Uighurs more strongly than Han, as a hypothetical.
A virus “optimized for genotypes” is just about the stupidest thing one could think of.
Because as soon as you release it, the virus is under selection pressure. It will very very quickly mutate to spread as widely as possible, since that by definition is a huge fitness advantage over the originator’s ethnic preferences.
Would be a total “leopards ate my face” technology
They’d vaccinate their own citizens first, obviously. No idea why they would (allegedly) experiment with corona viruses for this as mutability higher than other options.
I remember being told by an epidemiologist I worked with on a Covid project in early 2020 that he doubted any C-19 vaccine would provide lasting immunity given the structure of the virus (he had worked in HIV vaccines).
It’s still a world historic dumb idea. The vaccinate the citizens idea quickly runs into 1) Illuminati-levels of conspiracy and complexity to vaccinate an entire 1.4B people without anyone else getting a sample and being like “why does this Chinese flu shot have… not flu… in it??” 2) when the virus spreads worldwide no one notices “hmm… China is having a totally different experience here… and Chinese nationals aren’t getting sick” 3) inevitably, escape mutants overcome the vaccine in China anyway (though most people are protected against severe disease).
“Lasting immunity”, what does that mean? Immunity against COVID-19 infection, no one thought this was easily achieved. Immunity against severe disease? Different story. Coronaviruses are hit and run, take over your nose for a few days to quickly spread, then the adaptive response usually knocks down the infection before lung damage or systemic infection. Thats the “equilibrium” between coronaviruses and our immune system. And that’s more or less what happened.
These are the kinds of "expert" assurances that I think Noah is concerned about, and I agree with him. All of your arguments are based on the current state of knowledge and technology, and seem to ignore the accelerating rate of technological development. For example, your point that gene synthesis isn't cheap because it costs a "few hundred bucks for one block". What did it cost ten years ago? Now project that collapse of cost forward even a few years. As someone who has spent the last ten years of my professional life building AI platforms I think you are massively underestimating the impact of the rate of development. The risk Noah refers to may not be viable today, but his fear is completely valid because, if we wait until it is viable to worry about it enough to focus real resources on mitigating it, it will be too late.
I am - in the near to mid term (say 3-5 years) - more worried about rogue nation states and terrorist groups becoming much more sophisticated at creating biological and chemical weapons than the angry teenager who wants to go out in a blaze of high media attention glory (incredibly bad as it already is)…because they are already committed to developing WMDs. The skill required to create bioweapons has been falling for decades, even before LLMs, and so this is something governments and security agencies have been monitoring for a long time. But it is hard for all the reasons pointed out here by John. As a first order suggestion, let’s get much better at early detection/warning on pathogens (both natural and synthetic) via wastewater, passive environmental sensors in high volume public settings, etc. Let’s work more on broader spectrum antivirals, rapid vaccine deployment, and AI systems whose job it is to monitor the social networks where this kind of planning or behavior is likely to germinate. I’m also more worried about open source or jail broken LLMs that run locally and basically impossible to monitor. As Noah points out, AI safety advocates are focused on risks that are 1) easily understandable (jobs, cybersecurity), 2) most politically salient (jobs, misinformation, environmental), and 3) aren’t so catastrophic and inevitable that people paradoxically shrug them off. It’s a challenge that climate activism struggles with - instead of saying we’re all doomed, start with resilience and mitigation and looking for solutions that piggy back on other popular/good policies. A global AI biosecurity treaty would be hard to enforce, have more holes than Swiss cheese…and still be a good place to start because it sends lots of money and smart people to work on detection and mitigation.
Great point, a terrorist cell could do this even easier than a teenager. And they will have a lot more resources to throw at the problem, including if needed kidnapping family of scientists to force them to perform work
Well, this is definitely reassuring news from a real expert! Here’s how I’m digesting all of this—see if I’m tracking correctly:
1) Massive power requires massive complexity: Any technology with true WMD-level destructive power remains complex by nature; it’s never going to be cheap, trivial, or accessible at the push of a button.
2) Physical bottlenecks still exist: AI doesn’t operate in a vacuum—it still relies on sophisticated, strictly regulated physical laboratories and, crucially, skilled human hands.
3) The defense holds the upper hand: The overwhelming majority of people with these specialized skills are doing constructive work. While there might always be a rare lunatic dreaming of world destruction, there are far more experts like you keeping an eye out to intercept them.
4) The ecosystem’s natural balance: Just like in any specialized field, we have a vast majority of productive professionals building for the common good, a tiny handful of bad actors, and a much larger, better-equipped team of "police and detectives" keeping tabs on the risks.
So overall, Noah provides a useful, sci-fi-esque warning to keep us on our toes, while you provide the grounded, reassuring reality check. Is that a fair takeaway?
As someone coming from a software engineering background rather than biology, I hold a similar view on AI-driven cybersecurity threats—like automated mass network attacks or systemic privacy exploits. Given that the world has far more dedicated biosafety experts like yourself than malicious rogues, I'm inclined to think we are actually in a much safer and more optimistic position than the doom scenarios suggest.
I will got deeper on "the ecosystem's natural balance". Viruses are as old as life itself. The battle between viruses and immunity has been going on for billions of years, in trillions upon trillions of infections, each with selective pressure on both the virus and the host. Our immune system is an absolute marvel, on par with the nervous system in complexity and subtleness. So are the viruses it fights; packing just enough genomic information into a tiny nanomachine, to temporarily evade the immune system enough to rapidly produce and release sufficient nanomachines that can survive several brutally hostile environments to get to the next host.
The factors that a potential AI bioweapons maker cares about are largely incidental factors shaped by those billions of years of arms race. They're largely opaque to us, with extraordinarily limited data. So they're going to be incredibly hard to artificially engineer.
Which leads to a 3rd point I didn't bring up: "alignment". As soon as that virus is released, is de facto becomes "misaligned" with the weapons maker because it is immediately going to encounter selective pressure. That long incubation period? Maybe some mutant shortens it 2X and gains a transmission boost with an incidental 50% decrease in morbidity. Devastating morbidity totally annihilating an early infected region? Oops, a much less morbid variant, re-infecting variant just dropped and is now spreading outwards. Selection is going to push the virus in all kinds of unpredictable directions, and immunity is then going to push back, all in ways that very, very quickly overcome any engineering.
None of this means we shouldn't add biosecurity to AI harms; I think it's a very wise idea. I just think the scenarios people assume far more about our ability to engineer the unpredictable than is warranted.
It's fascinating how the tech world borrowed the biological term "virus" back in the early days of Windows and the nascent internet. The engineering hurdles in both domains share almost the exact same logic. People often forget that a computer virus is still a meticulously written program: first, it has to be syntactically valid to even run; second, it needs a functional transport vector to replicate via networks or storage; third, it must stay stealthy enough to evade antivirus scanners; and only then can it attempt to execute its malicious payload. A poorly engineered computer virus suffers from its own "misalignment" problem—smashing random keys on a keyboard won't yield an executable that crashes global banks or airports. The barrier to entry for devastating harm remains immensely high.
Of course, with AI in the mix, more bad actors might think, "Hey, let me give it a shot." But this quickly devolves into the classic cat-and-mouse, spear-and-shield dynamic. While cybersecurity doesn't have billions of years of biological evolution behind it, it is a fierce arms race nonetheless. And because the technical capacity to build both destructive malware and defense systems remains concentrated within a relatively small subset of skilled individuals and institutions, effective oversight and mitigation are entirely doable.
Drawing this analogy really helps me grasp your point across disciplines. To be honest, recognizing these shared principles makes me feel much more at ease. As a software engineer, I've never really lost sleep over Hollywood-style hacking where someone effortlessly breaches high-security systems with a few keystrokes—even in the age of AI-assisted coding, which I use daily as a decent developer myself.
True. Returning to the police vs. criminal analogy, each side has its own upper hand: the police possess power, resources, and enforcement tools, while bad actors leverage flexibility and stealth.
This endless cat-and-mouse dynamic continues. AI has undoubtedly given bad actors fresh avenues and a temporary edge in both cyber attacks and biosecurity. The crux of the issue is for defense and governance to rapidly adapt and restore the balance—which is exactly why this post and comment section are so constructive.
Not that there is zero chance of AI (or anyone) designing a doomsday virus, but I don't think there's an example to start with of a "very contagious viruses that have very long asymptomatic contagious periods and very high mortality rates once they become symptomatic". Organisms have evolved to recognize non-self pretty quickly, and pretty good defenses against airborne attacks, which is why 30% is the highest mortality rate for untreated airborne viral illness. Contact viruses can be more deadly, but have limited levels of contagion. I agree that a virus is one most likely route for societal breaking, but compared to the evolutionary space being explored by viruses every day, all the foreseeable AI is a drop in the bucket. When I was in grad school we often said the large numbers would be more accurately described as biological rather than astronomical.
Honestly, I think a lot of people here played too much Pandemic 2, that old Flash game where you could select your bug's traits at will, and then (inexplicably) "update" it as it spread and you gained more XP (and also inexplicably, there was no spontaneous recover/immunity... people had your bug until a vaccine or a drug came along). The key was keep symptoms non-existent until the virus reached Madagascar and New Zealand, then turn on the hemorrhaging and strokes!
Good article, Noah, and I completely agree this is the most likely doomsday scenario and not getting enough attention. For those reasons, I would highly recommend you make this a free article so that it can be shared and read more widely.
6. Since we live in a world with AIs that can create bioweapons the biolab, which is not staffed by angry teenagers who want to release bioweapons, has access to the same AI (probably better) as the angry teenager does and runs the incoming sample requests though the standard "is this a bioweapon" prompt.
Imagine this with human virus designers and human virus checkers. How easy would it be to identify a dangerous utterly novel pathogen from just seeing the genome?
Obviously AI can throw more compute at the problem. But will every biolab do so?
I think you've got the general shape of one piece of the solution right, but there are a lot of obvious scary holes in this safeguard!
Typically in this nature of problem the attacker has the disadvantage because the problem of "design something that will do X" is fundamentally more complex than the inverse "will design X do Y?" This seems similar to "design an airplane that works" vs "look at this airplane design and tell me if it can fly." I'm not an expert in this field, though, so there could be something lurking out there that actually makes the attacker's problem easier than the defender's, but I have never heard an argument for that.
As for the non compliant biolab problem I think that's pretty easy. The screening tools can be publicly funded infrastructure. We can even pay labs a bit of money to send their samples in for screening. This means there is absolutely no incentive not to comply.
The problem is much closer to "order a set of 3D printed parts that can be used to make an airplane" and "predict what this catalog of 500 printed parts assemble into". I think it's very offense dominant.
Fallingknife beat me to the punch. I also think he phrased it too strongly, but the jist of what he wrote is raising a legitimate point. Noah, even underground labs won't want to design killer viruses and whatnot, that's bad for business and for themselves. Have you considered this?
To be clear. I don't think it's impossible for a lab to be so poorly run that it would not conduct a basic AI check before making a a custom made virus, but I'd also expect the correlation between a lab's ability to make a virus and do a (what seems in your hypothetical) a fairly elementary check that can be automated to be pretty negative. If they don't do this, cheap and easy enough that an angry teenager can do it, they're probably not the best and may not be able to make something tricky.
This doesn't eliminate the risk, far from it, and I think you're highlighting this risk correctly. But you may be overestimating the probability here. It may need someone capable of using a lab themselves...
Also. If The Economist is writing articles/leaders on this, it's surely not so far from people's mind at this point.
Additionally, the assembly could be done in house, and many gene synthesis companies do not screen orders. It's worth noting that bioterror is very rare, and nobody has ever seriously tried with a contagious agent. Thus, in cut-throat low-margin businesses like DNA synthesis services, it makes sense to cut corners here.
Really, what would be the odds that YOUR company would fulfill an order to take out half the world before 2028, given the historical evidence is so weak? I don't agree with this view, obviously, but it's not crazy.
I'm not saying it's impossible, sure, but the hypothetical here is an AI that a not-genius-level teenager is able to use to design a few superviruses successfully. Let's add the necessary premises that I think are needed for this to be plausible, feel free to disagree here.
- Jailbroken, super powerful AIs are not too hard to acquire. (The skill level for a teen to do it would surely match that of a lab)
- These AIs are not very expensive to run. (Teens typically don't have huge resources, ditto the labs)
- These AIs should be pretty good to both design and figure out what a virus does. (It goes without saying, trying to hide characteristics would seem tricky for another AI to detect)
- Underground labs have some significant incentives to not create things that draw attention to themselves and/or kill themselves. (How much is debatable, of course, but the incentives are there)
It only takes one slip up, one badly run but still capable of making superviruses lab, of course. But it probably lowers the risk, to some extent.
So number 1 is true now and will remain true as long as open-weight models keep getting deployed.
Number 2 is probably true. As someone else mentioned, the main cost would be reagents. We're probably talking about a budget of 50-100k, of which a few thousand at most must go to compute. I think a teenager is a bit of an exaggeration here, but not outside of the realm of the possibility.
Number 3 is not needed (for the most part) on the enhancement side because of previously published information. On the obfuscation side, it's likely offense dominant. AIs do have to become better to do this reliably, but the situation laid out in the original post is achievable without superintelligence.
Number 4 probably won't involve underground labs. They'll be legitimately fulfilling orders for pharma companies and small research labs. Some labs will figure out that if you don't screen any orders, you can shave off a few percentage points from order costs. This is a fairly easy coordination problem to solve, but we are not track to solve it due to the low salience and speculative nature of the risk.
To keep things brief. If your assessment of 2 is correct, that does change the calculus further, but decreases the probability a bit more, in my view. Still too high to feel safe, but lower.
3 Your point about the offence might be true, but it seems a bit speculative to me, I'm honestly not sure... and yes, agreed, I'm not discussing a superinteligence scenario here, sorry if my writing (which was a bit hasty) gave that impression.
4 I'm not sure. A few percentage points would be quite huge! But let's put it like this. If I understood you correctly (apologies if not) and the reagent costs are 50-100k, taking that as the baseline cost for manufacturing the viruses, then you're saying that the cost of using AI to screen would be at least 500-1k (that's 1% of the total cost). Doesn't that seem too large? If it's less than 1%, we might still have issues (depending on how competitive the market gets!), but it does seem like a silly thing to skimp out on.
I mean, look. Even then, processes are not perfect and even if I'm right, a lab would need to make one mistake with one rogue agent for everything go wrong... so I'm not doubting the existence of the risk. I'm just feel that this point reduces the probability significantly.
For 4, the providers have to screen everybody, so the cost applies to their total revenue. It really depends on the set up, but to give you a sense of cost, you have to run sequence alignment for every fragment you get and then have a human reviewer verify the flagged ones. I think today, a few percent of gross revenue is accurate, but many of these costs go to research and development of the pipelines. Marginal cost is probably more like a tenth of a percent of the cost of the order. But it could get worse if you insist on stronger methods to beat stronger offense.
Like I said, I think this is a crazy thing to skimp on, but as the industry grows, there will be a lot of competition. Many people just completely dismiss these risks out of hand, so it's likely you'll get a handful of companies that don't screen. Then you just have to identify them with some probes. This applies to both DNA synthesis companies (which exist now and has many examples of non-screeners) and end-to-end cloud labs (which is nascent and could go either way).
Or your pissed off teenager is instead a pissed off graduate student at a research university, which means he can find out where the biolab is and has an excuse to be there...
Fair point even if it's not a trivial proposition.
That said, it does raise the question how likely it is for a terrorist organisation to want to create a unlimited reach bioweapon. I don't think most terrorists want to see the total human destruction, unlike the more nihilistic viewpoint Noah proposes.
Still, not an impossible scenario, no, and worryingly so.
This is easy to get around. Order pieces of DNA that look harmless in isolation then stitch them together yourself into something dangerous. Which not everyone can do, but some people can.
Sadly, I must agree with Noah on this one because it's not only the angry teenager but also rogue states (North Korea) and a long list of terrorist organizations that may make this effort. That the downside risk is mass death or civilizational collapse means even a tiny probability of this event must be taken seriously. Given the track record of the human race I've decided to front load my retirement "bucket list". In my 69 years I've never experienced this level of uncertainty about not only my future but all our futures.
A version of this occurred to me many years ago as an explanation for the Fermi Paradox: That intelligent, technology-oriented life forms only survive until they achieve a level of technology in which any ordinary individual in their society has access to technology capable of wiping out the species. With large enough population size simple variability in individual behaviour dictates that individuals will periodically attempt world destruction--I have seen limited efforts in this direction in the news multiple times throughout my life: Jonestown mass suicide, the Japanese attack you mention, the Manson "family", and more recently Putin threatening suicidal nuclear WWIII if he isn't allowed to crush Ukraine, etc. Once that level of destructive capability exists, it's just a matter of time until someone succeeds. If we look at the destructive capacity of an ordinary individual human, it has been increasing at an accelerating rate for some time. I don't think we are there yet, but I agree that the reasons I see even--especially--experts presenting to dispel alarm actually just increase my alarm because they always seem based on the level of capabilities we have today, while ignoring the accelerating rate of technological development. I'm 74. The things I can do today from my computer, or even just using my phone, would astound my 24 year old self. One of two things will happen, and probably within the next decade: We will either figure out a way to modify or constrain human behaviour, or human behaviour will destroy us all once that ordinary but disturbed individual has access to the capability to wreck that level of destruction.
I come to the same conclusion. We will soon be able to annihilate the human race from the actions of a few bad actors. This is true with or without AI. It is virtually inevitable.
Thus our ONLY hope is that AI can prevent us from doing so. We need a new higher level of intelligence and morality than what the current naked apes are capable of alone.
What struck me is the weakest link: some humans are evil. A line from "Hi Ren" starts playing in my head:
My name it is stitched to your lips, so, you see
I won't bow to the will of a mortal, feeble and normal
You wanna kill me? I'm eternal, immortal
I live in every decision that catalysed chaos
That causes division
I live inside death, the beginning of ends
I am you, you are me, I am you, Ren
I'm dismayed at the senseless destruction caused by wars that seem to arise from human greed: the lust for power, money, land. Gaza, Ukraine, Uganda, WWII, ... If we cannot defeat the immortal forces of evil, then Fermi was right.
The extreme recklessness of teenagers is what makes it plausible that he would want to do this, but other things about teenagers militate against him succeeding: there's lots of stuff teenagers don't know, they tend to lose interest in things fairly quickly and have little or no experience at negotiating their way through complex processes. This would be nowhere as easy as breaking into your Dad's gun cabinet. His plan would very probably come off the rails at some point in the following parts of the process:
- finds an unlicensed biolab in East Europe - these are probably not as common as we fear, plus, searching for them online, even via AI, is likely to raise some sort of red flag somewhere in the surveillance community.
- that ships genetically modified viruses - i.e. a supplier that has little interest in their own survival which is very rare
- It . . . makes a potential doomsday virus - the teenager would probably have to have a lot of money (for a teenager) to make this happen and there might be no such thing as a doomsday virus or at least the gate through which any such virus has to pass might be very narrow relative to the design space that has to be explored
- ships these to the angry teenager - again, they would have to have little interest in their own survival
- The teenager mixes the samples together - i.e. without inadvertently destroying them, reducing their potency etc. Rogue biolabs are notoriously remiss when it comes to providing instruction manuals and customer support.
There will be any number of very mundane things that can go wrong for the teenager at each of these points. So getting through each of them alone will be improbable, and each of these low probabilities are fairly independent of each other. But in order to succeed, the teenager has to have all of them happen. That's a lot of low numbers multiplied by one another. I'm not saying it couldn't happen - almost anything could happen - I just think it's extremely unlikely.
It's at these rather workaday points in the process that regulators and governments need to position their roadblocks, to make what is already difficult, well-nigh impossible.
Yes, but these objections aren't about efficiency in search and information processing or effectiveness in problem-solving - the kinds of thing that AI can do. They are about real-world probabilities (e.g. rogue, affordable E. European biolabs being rare) and hands-on expertise: the sort of things that you can only learn by practice, seeing and doing repeatedly. Even at undergrad level chemistry students spend a lot of their time in the lab, learning their craft. There would probably also be implicit knowledge barriers to finding, contacting and ordering from the rogue labs: criminal enterprises don't tend to advertise themselves as such and often prefer word-of mouth referral and face-to-face communication to avoid being detected or leaving an evidence trail.
AI still has a long way to go before it can replicate and supply these types of implicit knowledge. And getting this type of knowledge would probably require AI to be embedded in physical devices which are far easier to track and regulate.
You are being too specific. NS outlined one possibility, reckless teenager. Here's another: bitter suicidal biologist at sketchy biolab (doesn't need to be criminal lab, just sloppy and lazily administered).
Maybe this case is also a matter of multiplying small numbers, but the number of cases might be large . Multiply small numbers enough times and you get a positive.
Psycho biologist in small nasty nation's newly feasible program. Same nation's biolab program but the antidote is faulty. Same scenario but the antidote works and the dictator is crazy enough to act. I could go on.
The main point is capability. If AI makes work that previously required large coordinated teams feasible for an individual you've got a problem. Intelligent psychopaths exist, the AI itself doesn't need to turn on us.
Would the psycho biologist need AI? My point is that with people who are clever but not mature, experienced or socialised enough to have any accountability whatsoever (and so are prepared to take the whole world with them), you start with very low numbers which then get thinned out rapidly by hurdles, which they face because they're working alone and are insufficiently worldly to know how to deal with them, each of which stops all but a very small %. If the point is that a rogue state or well-organised death-cult-style terrorist group can with AI now, for the first time, manufacture biological weapons that are so deadly they could wipe out the whole world as we know it - that scenario runs into the objection that there are staggeringly few such groups or people. Not wanting there to be a world anymore and being able to do what's necessary to bring that about, even with the assistance of AI, are in serious tension. The overlap between the two is very small, potentially non-existent. It's certainly a risk, but in pretty much any telling of it that I can imagine, it requires so many moving parts that success is incredibly unlikely.
The current leading edge science is team work. An extremely gifted biologist might create one virus, but not 100. This involves different steps (design, prototype, multiplication, introduction to the wild) that would likely be separate specialties. All those "hurdles caused by their being alone" are quite easily overcome with a sufficiently capable AI.
It is the current paradigm that you need teams for everything that matters and assembling a functional team of unstable psychopaths is nearly impossible.
When AI empowers the individual, that safety is gone.
See my response to Noah above. Most of the hurdles created by being alone are not easily overcome because they're not a matter of having more hands but of having hands that have acquired implicit skills and judgement passed down from previous practitioners and that are connected to heads with contextual knowledge and bodies that can win the trust of bad actors with highly suspicious natures. AI will, at the very least, take time to duplicate these and may never be able to, at least not in a way that regulators wouldn't be able to keep pace with.
The recent robot Olympics in China provided a glimpse into this issue. In the races, the robots were very good at starting and running but slowing down whilst going round a bend in the track and avoiding other runners - something that human sprinters do with effortless ease - is immensely complex. So their creators didn't even bother trying to make them do it. They just let them run into crash mats, often wrecking themselves in the process.
I think that for the foreseeable future our teenager's plan for world destruction would also probably end up amongst the crash mats for want of a range of implicit skills that AI can't mimic.
As long as you picture a teenager, I tend to agree.
But once you have a disgruntled scientist in mind, things change. A single person with good judgment and implicit skills combined with the leverage of AI could wreak incredible havoc.
Yes, that's true. But then such a person could do a fair bit of damage before the advent of AI. Now maybe they could do more or do it a bit more easily or there could be more people who could stretch their expertise across into areas of threat than there were previously. But it's not the transformation of the risk landscape that it would be if a smart teenager could pull it off. And some of the barriers highlighted would remain even for a skilled, experienced scientist.
It's quite possible that we now have stricter regulation, better coordination, and more transparency *among virologists and their regulators*, but done on the quiet by the insiders in virology research. But nobody on the inside has an incentive to be more transparent *to the general public*.
You need intelligence agencies to flood both sides of the mail order custom pandemic market with honey traps. Today, if you try to find a community online that will help you be the next McVeigh, you will very likely wind up talking to the Feds.
Should be the same for would-be bioweapons. There should be ubiquitous crowds of fake buyers that trigger a visit from agents or drones to anyone who accepts their order. And fake unlicensed labs that trigger the same to any would-be buyers. Search is a hard problem, and fortunately it’s easy to find out where the AIs will look first.
There might well be already. In regard to nuclear weapons, I've read that it's quite legal to buy krytons, which are needed to trigger nuclear explosions, though trying to do so will get you a lot of attention from the feds.
I’m sure the feds monitor a great many things that would be useful in nuclear weapons production, but the truth is a basic nuclear bomb isn’t complicated or hard to make. I learned the science behind it in second year of a physics degree, and after that you need the materials and some reasonable precision engineering.
The hard part is getting the enriched uranium or plutonium. I’d expect that the fissile materials and things like centrifuges for making them are what are really heavily monitored.
You do have to get good enough at both alignment and international coordination that your fake buyers and fake sellers don’t wind up trying to arrest or drone strike each other, which is non-trivial.
How exactly are the models going to improve to the point where they can do this? The reason AI has improved so dramatically since 2023 is largely due to quotidian software engineering around harnesses, not breakthroughs in model design. The models themselves are still transformer-based models, and the improvement has largely been in scope rather than in kind—i.e. bigger context windows, better harnesses, and efficiencies at inference.
Claude Code is magical for computer code because the internet provides extremely robust training data for extant coding languages. But, as the molecular biologist mentioned, relevant open-source biological data is much sparser and spottier. The operation being imagined here would also be more or less unprecedented in human history, so the data on how to actually execute such a plan is sparse as well. An LLM asked to fill in those gaps would likely default to fairly crude works of fiction.
This matters because I am dubious that skill in hacking will simply extend to skill in bioterrorism. There is a lot of data on how to hack successfully and not much on how to do bioterrorism—or, for that matter, how to do large swathes of work in the biomedical field. The analogy between an AI becoming an extraordinarily capable programmer and becoming an extraordinarily capable biologist therefore seems much weaker than the article assumes.
And the problem is not merely figuring out how to do something unprecedented. It would also need to figure out how to evade detection in the process. Details on how specific surveillance systems work are themselves unlikely to be comprehensively represented in easily accessible training data. The model would therefore need to use its imagination both to figure out how to actually execute the plan and to anticipate systems and circumstances for which it has little or no precedent. Modern LLMs are not reliably capable of doing that. I have never once encountered a useful hallucination. In fact, it feels much more likely that a layman doing something so complicated would end up in a blind alley after the model took a hallucination and sprinted off the reservation with it.
If we assume the progress in generative AI continues as it has since GPT-3.5 into 2029, the villain here will be using a model which still has all these shortcomings. Throwing more compute, making context windows larger, making tokens cheaper etc will only be diminishing returns if the scaling curve is sigmoidal rather than exponential, and there is good reason to believe that it is, and has been for ~18 months now.
What we’re talking about is an AGI capability here, and I don’t think the current approach of LLMs in harnesses are AGI. To get there, the labs are clearly going to need another breakthrough on par with Attention is All you Need. I won’t profess to know what breakthrough gets us to AGI—Yann LeCun says it’s world models, which is as good a hypothesis as any. Sam and Dario hand wave this problem away by saying their models will magically find this breakthrough on its own. This is again something that would require an imagination their systems are structurally incapable of.
So the part of the 2029 scenario I find unconvincing is not the premise that AI capabilities will continue improving. It is the assumption that continued improvements of the kind we have seen since GPT-3.5 necessarily produce the qualitatively different capabilities this scenario requires. The doomsday-virus argument quietly assumes that scaling and better agentic scaffolding will somehow turn a system that is exceptionally good at recombining abundant precedent into one that can reliably solve a sparse-data, unprecedented, long-horizon problem in the physical world. That is precisely the leap that needs to be demonstrated rather than assumed.
"quotidian software engineering around harnesses" is not true.
Post training for in harness agentic behaviour is the key. You can test it right now: put an early model in a harness and try and build something with it. It won't be pretty.
Without getting hung up on what is quotidian, I don’t think the way they’ve improved post-training is a fundamental breakthrough rather than an in-kind improvement on their existing methodologies. Which is to say they are iterating on the software pipelines they’ve built to crank out these models. More to my point, the improvement in agentic systems is because these companies are using the biggest pots of money ever to specifically be good at using these harnesses for software projects. I don’t believe they are doing so for biology, and if they are doing so, they absolutely should not be.
A lot of people are working on real learning and real understanding. One approach is Accelerated Understanding Inc.
You are correct, that you need more than just a LLM with sufficient scale. But building this "more" is exactly the current frontier.
Since current AI makes experimentation and iteration vastly cheaper and easier than in the old days and really bright minds provide the direction and philosophy, I don't expect it to take very long.
Two quibbles: You are vastly underestimating the damage a nuclear war would cause, though it would certainly solve the global warming problem for a while. Estimates are that 99% of those living in the northern hemisphere would die, the survivors from starvation. There may be fewer bombs then the Cold War days but they are much much more destructive.
And in the bioterror scenario, with much less physical destruction, I suspect a survival rate of 10 percent or 800-900 million people with access still to libraries could manage to live significantly better than a Neolithic lifestyle.
Smaller bombs are not less destructive: "multiple smaller weapons are more effective than a single larger one. Twenty 50-kiloton warheads, for example, destroy nearly three times the area leveled by a numerically equivalent 1-megaton weapon." MIT report
Don’t forget the EMP effect of nukes. A nuclear war is likely to open with a volley that includes using EMP nukes in orbit to wipe out satellites en masse. That would also knock out electronics on the ground in a massive radius.
Also, I agree with tomtom below. By all accounts, the known nuclear stockpiles are smaller warheads than early to mid Cold War because many smaller warheads are more effective than one big one (due to the inverse square law of energy dissipation).
No one can regulate what they don’t understand. Especially the caliber of men we blithely elect to run our country. This has happened over and over in our history with technology, science and invention. Private corporations end up doing the unregulated research and introducing products to make profits often with potentially stunning consequences. Take forever chemicals and genetically modified crops for example. I remember 60 years ago when effective genetic manipulation techniques were discovered and went unregulated for years. The same is happening now with AI. We have always lived with this problem like hopeful, trusting lemmings, and one day it could easily destroy us. It won’t be man’s intelligence that kills him but his ignorance and fear of what and whom he does not understand.
ARPA-H seems to be working on a plausible mitigation effort: an indoor air-monitoring program that aims to create "novel biosensors and risk assessment technologies [that] are designed to detect and respond to airborne biological threats earlier than ever before and to build a future where healthier buildings lead to healthier lives."
A small wrinkle to add to your scenario: even if a vaccine is created, some people might refuse to take it due to the effects of vaccine misinformation from the Covid era. (In my opinion, that is going to make any future pandemic worse, and thus make pandemics a more effective way of killing people).
By the way, recommend Station Eleven as interesting on how a population devastating virus might play out.
Molecular Biologist here. Two points:
1) #3 “it’s really hard” deserved more attention. Simply put, it’s really hard in exactly the ways that are very difficult for advanced AI to get around. The “training data” is extremely sparse; we have very little knowledge of what factors determine things like incubation periods, contagiousness, mortality and morbidity. We especially don’t know enough to predict with accuracy how changes needed for factor affect all the others. Advanced AIs can try to extrapolate from what we know, but those are likely yo have a HUGE dud rate without at least a few million (likely much more) spent on testing, optimization, iteration. Committed governments and terrorists could do it, but it’s high bar.
1a) gene synthesis ain’t cheap. Will set you back a few hundred bucks for one block, need several to get a viral genome. Oh and you usually need somewhat specialized equipment and knowledge to actually get DNA into cells and QC its doing what you want it to do. Costs quickly add, won’t deter those with deep pockets but does set a bar in who’s in the game
2) Infectious vaccines. You heard that right. In a world where AIs “understand virology” to the point can readily design a highly contagious, long incubation period, high morbidity virus, and other AI can just as easily design an extremely fast, contagious and benign virus, and then just combine in the needed (AI-optimized) mRNA vaccine components.
Release it, it spreads faster then the doomsday virus, no manufacturing, supply, distribution, or consent.
Take that RFK Jr!
Thanks!!
Note that if everyone already has the virus by the time it starts killing people, then even an infectious vaccine won't do much. You'd need an infectious agent that produces large quantities of antiviral in the body.
Also, sure you don't know 100% that a doomsday virus will work, which is why the terrorist releases 100 at once!
Also there are even scarier things that I didn't want to write explicitly about, but some biosecurity people have been telling me about. Happy to talk on the phone sometime if you want!
"You'd need an infectious agent that produces large quantities of antiviral in the body."
Pretty much every vaccine in existence does this! Within a few days of a vaccine (or naive infection), the B-cells that recognize viral proteins ("antigens") divide rapidly and secrete exponentially increasing amounts of IgM antibodies that bind to the viral antigens. These antibodies don't just prevent infection, they can block the spread of virus between cells in the body and target infected cells for clearance. A little bit later on maybe a week or so, T-cell response kicks in too. This produces "antivirals" in two broad flavors: helper T-cells that further stimulate the production and maturation of antibodies, and killer T-cells that specifically seek and destroy infected cells (and can effectively recognize ANY part of the virus, even "hidden" ones that evade the rest of the immune system).
So after about 2-3 weeks, even if this doomsday virus managed to produce absolutely NO adaptive immune response in its "dormant" phase (itself a super-hard problem), a vaccine or "benign" virus would likely kick off a highly effective adaptive response that greatly improves survival when the "morbid" phase activates. And if instead the virus did provoke a even a weak, low-level adaptive response in dormancy, then the vaccine would act like a booster shot, where exponential B- and T-cell activation both occur all over again, but in a much more coordinate and mature way.
"Also, sure you don't know 100% that a doomsday virus will work, which is why the terrorist releases 100 at once!"
Odds are only a few of those spread widely, even if all are highly transmissible. 1) each individual virus spreads separately from the other 99, because you cannot make a massive genome that encodes all 100 variants fit inside a tiny virus, so each very quickly comes under its own selection pressures. Let's say the avg in that 100 has an R0 of 4, and the best is just 20% better, at R0=5. After 10 infection cycles, the most transmissible one spreads 10X farther than the one average ones. This also assumes no cross-immunity between the 100; if there is even to a small degree then the 100 end up competing with each other and you get a few winners.
Vaccine design doesn't suffer from this problem, because what matters are the handful of very short sequences that produce the strongest antibody/T-cell responses. Even pre-LLMs, this could be predicted a-priori from sequence to a fairly decent (now excellent) degree, and can be confirmed very quickly (weeks) by analyzing serum/white blood cells from a few dozen patients. You have plenty of genomic space to daisy chain 100 or more short but highly immunogenic sequences into one backbone, ensuring that all 100 "stay together".
"Also there are even scarier things that I didn't want to write explicitly about, but some biosecurity people have been telling me about. Happy to talk on the phone sometime if you want!"
Thank you, I am sure there are much worse scenarios I can't think of, but I prefer to sleep at night : )
And FWIW, I commend you for thoughtfully calling attention to AI bioweapons as an under-appreciated high level risk. I absolutely share this concern. I just think it's "hard" in a way where more "intelligent" AI doesn't dramatically elevate risk until you get to VERY high levels of both automation and intelligence. But all long-tail risks are "hard", and biosecurity should be higher on people's radars.
I agree with every point that Froberg has made on this thread, but I wanted to chime in to emphasize the evolution point. Once you release this AI designed superkiller, it immediately succumbs to ordinary selection pressures. If the traits that make it super lethal are not adaptive, it will quickly lose them in a few generations. At that point you don’t have a superbug wiping out humanity, you have a tragic but very limited bioterrorism event that kills a few dozen people.
The cleverness of AI really comes into play less than the initial scenario supposes. The selective pressures the AI superbug encounters are emergent phenomena that the AI cannot know in advance because they are largely unknowable.
Isn't "long period of asymptomatic contagion" also quite hard? Contagion requires the virus to replicate a lot of itself, but it's replication and the body's immune response that causes most of the symptoms.
I think instead you'd need a virus that produces mild cold/flu symptoms in its early stage, when it has replicated a lot on the body, and then perhaps goes completely dormant for a few months (somehow) and then comes roaring back in some Ebola-like organ-shredding mode.
Either that or a virus that somehow manages its replication so it reaches a barely-contagious but sub-symptomatic level and then holds there for a long period of time... and it would need to do that in everyone, in spite of wide variations in bodies and their immune responses.
I'm certainly no expert, though. Maybe this is easier than it seems? Are there any examples of naturally-evolved viruses that work this way (even without the organ-shredding secondary mode)?
Yeah, It’s not easier, to my knowledge most human viruses really don’t behave that way.
Incubation/dormancy periods have a fitness cost: energy/complexity needed to evade innate immunity or maintain a latent state when you’re not spreading much. Could have a long asymptomatic shedding period, with active replication but epic immune evasion, but that also takes a ton of energy.
Organ shredding mode also has a fitness cost: shred organs too hard and you wipe out your own host!
Put another way these traits are hard to evolve, much less both engineer and engineer in such a way that they don’t just get quickly mutated out into something more in the virus’ interests.
Untreated HIV does this, with the caveat that it was only transmitted by bodily fluids.
HIV does not spread quickly at all, compared with COVID, influenza, measles, etc. Airborne, droplet, saliva, fecal transmission all require infection in mucosal tissues, versus deep in the blood, and very fast replication of huge amounts of virus. Fast replication in mucosal tissues causes tissue damage (versus "silently" clearing out a fraction of white blood cells), tissue damage and fast replication of all trigger an innate immune response.
Put it another way, HIV has a highly unusual strategy of directly and persistently infecting immune cells in the blood, which is not easy to combine with rapid contagion.
So, not very contagious.
Creating a Timothy McVeigh-type mass casualty event is no longer easily possible, because public purchase is no longer possible of the ingredients necessary to make a truck bomb.
The FBI needs to be authorized by law to more closely restrict the sale and purchase of any and all materials that could be used to produce bakers dozens of vibecoded killer viruses.
McVeigh used ammonium nitrate and nitromethane, neither of which is substantially regulated. There is a voluntary reporting program so some vendors may report sales they deem suspicious, but that just requires potential bombmakers to find a vendor who won't report them, or structure their purchases as part of activities that make them unsuspicious.
So why don't we have an HIV vaccine that works yet? The natural antiviral agents the human body produces simply aren't able to get the job done in most people who get infected. It took the invention of artificial antiviral drugs before HIV turned from a death sentence into a life sentence.
HIV in the 1980s had nearly all the properties of a doomsday virus anyone could want; the immune system couldn't clear it, and it had an extremely long (mostly) asymptomatic period before its human host inevitably died. The only reason it wasn't an actual doomsday virus is that it could only be transmitted via bodily fluids. If HIV had been as contagious as measles back in the 1980s, we'd all be dead now.
HIV directly infects T-cells, key cells in adaptive immune response. T-cells themselves are an interesting target; silently clearing a portion of them out is a much weaker signal to the rest of the immune system than tissue damage in the lungs or gut. HIV also integrates into the host genome, leading to a large reservoir of "permanently infected cells". And on top of all this, HIV mutates at the very edge of stability, constantly evading the host's immune defense.
It might seem like a doomsday virus, but the key is all of these "odd" traits that make it so challenging for vaccine development/control without antivirals are in fact a coordinated set of traits optimized for long-term blood infection. That's a completely different set of traits than those optimized for high contagiousness and/or rapid spread.
For those viruses, T-cells are a poor choice for preferred cell because they have almost no "contact" with the outside world. Genome integration doesn't matter much if you're doing fast hit-and-run spread, and neither does extremely fast mutation.
Thank you for sharing your expertise on this subject.
I'm the opposite of an expert and don't want to be a pollyanna, but I just get the feeling that this is ascribing superpowers to AI that won't apply to the cussed complications of real world biology.
Which is not to say that the angry teenager couldn't kill 5% of the world population, unfortunately. Reason enough to proceed with great caution.
Taking rabies as an example of a disease that has a long incubation and nearly 100% mortality (luckily not contagious), the vaccine is normally taken post infection. So the conclusion that the vaccine won't work after infection is, at best, only partially true.
It seems to me that viruses constantly and furiously recombine themselves in almost every infected cell, so maybe millions of times in the span of a few hours for every infected living being. That's a massively parallel evolutionary search that has been constantly going on since before mammals walked the earth. It is true that, as you say, there are no particular evolutionary reasons for a virus to be deadly, but it's hard to think that an AI can outdo nature in exploring this space.
Why has nature never invented the wheel?
I think most of the answer is that smooth terrains are very rare in nature, and wheels quickly become ineffective when medium-big obstacles appear. I do think that if wheel locomotion was a survival imperative (and feasible biologically) nature would have invented it.
That said, I am not completely sure it’s feasible to grow wheels, axels and some kind of engine organically within a single organism. From a mechanical point of view. Maybe it’s possible as a symbiosis of different organisms.
So if your point is that there may be no natural evolutionary path that incrementally leads to a biological wheel locomotion system, while it might still just be possible to design it and assemble it, using engineered biological components, I guess that’s a good point.
Not sure how/if it applies also to viruses but it might. I guess the crux of the problem is that we can now design complex (biological) systems that nature has no reason to invent.
https://en.wikipedia.org/wiki/Wheel_spider
That's not a wheel; it's just a spider that tumbles downhill doing cartwheels.
A wheel, in this context, is an axle connected to a circular object (necessarily completely detached) used for locomotion. The flagellum comes close, but it's a detached "tail," not something round.
https://en.wikipedia.org/wiki/Flagellum
Depends on the virus’s incubation time and patterns. To give a very well-known counterexample, the rabies vaccine works even after exposure.
I agree that #3 deserves more attention. But I also agree with Noah that at some point someone it going to try this. My prediction is that the first attempt will be flawed and will not wipe out humanity, but will be bad enough that it gets people to implement the safe-guards Noah is recommending.
I can't say this is my HOPE, since obviously I can't hope for people to die. But this would be very far from the worst outcome.
All humans aren’t equally susceptible to infection or as likely to recover fully.
The Chinese allegedly were mucking around with optimizing viruses for genotypes. Military program. Say….infecting Indians or Uighurs more strongly than Han, as a hypothetical.
A virus “optimized for genotypes” is just about the stupidest thing one could think of.
Because as soon as you release it, the virus is under selection pressure. It will very very quickly mutate to spread as widely as possible, since that by definition is a huge fitness advantage over the originator’s ethnic preferences.
Would be a total “leopards ate my face” technology
You’d have to take it up with the PLA!
They’d vaccinate their own citizens first, obviously. No idea why they would (allegedly) experiment with corona viruses for this as mutability higher than other options.
I remember being told by an epidemiologist I worked with on a Covid project in early 2020 that he doubted any C-19 vaccine would provide lasting immunity given the structure of the virus (he had worked in HIV vaccines).
It’s still a world historic dumb idea. The vaccinate the citizens idea quickly runs into 1) Illuminati-levels of conspiracy and complexity to vaccinate an entire 1.4B people without anyone else getting a sample and being like “why does this Chinese flu shot have… not flu… in it??” 2) when the virus spreads worldwide no one notices “hmm… China is having a totally different experience here… and Chinese nationals aren’t getting sick” 3) inevitably, escape mutants overcome the vaccine in China anyway (though most people are protected against severe disease).
“Lasting immunity”, what does that mean? Immunity against COVID-19 infection, no one thought this was easily achieved. Immunity against severe disease? Different story. Coronaviruses are hit and run, take over your nose for a few days to quickly spread, then the adaptive response usually knocks down the infection before lung damage or systemic infection. Thats the “equilibrium” between coronaviruses and our immune system. And that’s more or less what happened.
Let's pray this is what happens, Sean. But, it's just as likely we'll implement half measures, call it good and move on.
These are the kinds of "expert" assurances that I think Noah is concerned about, and I agree with him. All of your arguments are based on the current state of knowledge and technology, and seem to ignore the accelerating rate of technological development. For example, your point that gene synthesis isn't cheap because it costs a "few hundred bucks for one block". What did it cost ten years ago? Now project that collapse of cost forward even a few years. As someone who has spent the last ten years of my professional life building AI platforms I think you are massively underestimating the impact of the rate of development. The risk Noah refers to may not be viable today, but his fear is completely valid because, if we wait until it is viable to worry about it enough to focus real resources on mitigating it, it will be too late.
I am - in the near to mid term (say 3-5 years) - more worried about rogue nation states and terrorist groups becoming much more sophisticated at creating biological and chemical weapons than the angry teenager who wants to go out in a blaze of high media attention glory (incredibly bad as it already is)…because they are already committed to developing WMDs. The skill required to create bioweapons has been falling for decades, even before LLMs, and so this is something governments and security agencies have been monitoring for a long time. But it is hard for all the reasons pointed out here by John. As a first order suggestion, let’s get much better at early detection/warning on pathogens (both natural and synthetic) via wastewater, passive environmental sensors in high volume public settings, etc. Let’s work more on broader spectrum antivirals, rapid vaccine deployment, and AI systems whose job it is to monitor the social networks where this kind of planning or behavior is likely to germinate. I’m also more worried about open source or jail broken LLMs that run locally and basically impossible to monitor. As Noah points out, AI safety advocates are focused on risks that are 1) easily understandable (jobs, cybersecurity), 2) most politically salient (jobs, misinformation, environmental), and 3) aren’t so catastrophic and inevitable that people paradoxically shrug them off. It’s a challenge that climate activism struggles with - instead of saying we’re all doomed, start with resilience and mitigation and looking for solutions that piggy back on other popular/good policies. A global AI biosecurity treaty would be hard to enforce, have more holes than Swiss cheese…and still be a good place to start because it sends lots of money and smart people to work on detection and mitigation.
Great point, a terrorist cell could do this even easier than a teenager. And they will have a lot more resources to throw at the problem, including if needed kidnapping family of scientists to force them to perform work
Well, this is definitely reassuring news from a real expert! Here’s how I’m digesting all of this—see if I’m tracking correctly:
1) Massive power requires massive complexity: Any technology with true WMD-level destructive power remains complex by nature; it’s never going to be cheap, trivial, or accessible at the push of a button.
2) Physical bottlenecks still exist: AI doesn’t operate in a vacuum—it still relies on sophisticated, strictly regulated physical laboratories and, crucially, skilled human hands.
3) The defense holds the upper hand: The overwhelming majority of people with these specialized skills are doing constructive work. While there might always be a rare lunatic dreaming of world destruction, there are far more experts like you keeping an eye out to intercept them.
4) The ecosystem’s natural balance: Just like in any specialized field, we have a vast majority of productive professionals building for the common good, a tiny handful of bad actors, and a much larger, better-equipped team of "police and detectives" keeping tabs on the risks.
So overall, Noah provides a useful, sci-fi-esque warning to keep us on our toes, while you provide the grounded, reassuring reality check. Is that a fair takeaway?
As someone coming from a software engineering background rather than biology, I hold a similar view on AI-driven cybersecurity threats—like automated mass network attacks or systemic privacy exploits. Given that the world has far more dedicated biosafety experts like yourself than malicious rogues, I'm inclined to think we are actually in a much safer and more optimistic position than the doom scenarios suggest.
I will got deeper on "the ecosystem's natural balance". Viruses are as old as life itself. The battle between viruses and immunity has been going on for billions of years, in trillions upon trillions of infections, each with selective pressure on both the virus and the host. Our immune system is an absolute marvel, on par with the nervous system in complexity and subtleness. So are the viruses it fights; packing just enough genomic information into a tiny nanomachine, to temporarily evade the immune system enough to rapidly produce and release sufficient nanomachines that can survive several brutally hostile environments to get to the next host.
The factors that a potential AI bioweapons maker cares about are largely incidental factors shaped by those billions of years of arms race. They're largely opaque to us, with extraordinarily limited data. So they're going to be incredibly hard to artificially engineer.
Which leads to a 3rd point I didn't bring up: "alignment". As soon as that virus is released, is de facto becomes "misaligned" with the weapons maker because it is immediately going to encounter selective pressure. That long incubation period? Maybe some mutant shortens it 2X and gains a transmission boost with an incidental 50% decrease in morbidity. Devastating morbidity totally annihilating an early infected region? Oops, a much less morbid variant, re-infecting variant just dropped and is now spreading outwards. Selection is going to push the virus in all kinds of unpredictable directions, and immunity is then going to push back, all in ways that very, very quickly overcome any engineering.
None of this means we shouldn't add biosecurity to AI harms; I think it's a very wise idea. I just think the scenarios people assume far more about our ability to engineer the unpredictable than is warranted.
The alignment problem—that's right & brilliant!
It's fascinating how the tech world borrowed the biological term "virus" back in the early days of Windows and the nascent internet. The engineering hurdles in both domains share almost the exact same logic. People often forget that a computer virus is still a meticulously written program: first, it has to be syntactically valid to even run; second, it needs a functional transport vector to replicate via networks or storage; third, it must stay stealthy enough to evade antivirus scanners; and only then can it attempt to execute its malicious payload. A poorly engineered computer virus suffers from its own "misalignment" problem—smashing random keys on a keyboard won't yield an executable that crashes global banks or airports. The barrier to entry for devastating harm remains immensely high.
Of course, with AI in the mix, more bad actors might think, "Hey, let me give it a shot." But this quickly devolves into the classic cat-and-mouse, spear-and-shield dynamic. While cybersecurity doesn't have billions of years of biological evolution behind it, it is a fierce arms race nonetheless. And because the technical capacity to build both destructive malware and defense systems remains concentrated within a relatively small subset of skilled individuals and institutions, effective oversight and mitigation are entirely doable.
Drawing this analogy really helps me grasp your point across disciplines. To be honest, recognizing these shared principles makes me feel much more at ease. As a software engineer, I've never really lost sleep over Hollywood-style hacking where someone effortlessly breaches high-security systems with a few keystrokes—even in the age of AI-assisted coding, which I use daily as a decent developer myself.
yes but the good guys need to be right every time, but bad guys only once.
And it could be a terrorist cell with millions in backing and a real lab instead of a teenager
True. Returning to the police vs. criminal analogy, each side has its own upper hand: the police possess power, resources, and enforcement tools, while bad actors leverage flexibility and stealth.
This endless cat-and-mouse dynamic continues. AI has undoubtedly given bad actors fresh avenues and a temporary edge in both cyber attacks and biosecurity. The crux of the issue is for defense and governance to rapidly adapt and restore the balance—which is exactly why this post and comment section are so constructive.
Not that there is zero chance of AI (or anyone) designing a doomsday virus, but I don't think there's an example to start with of a "very contagious viruses that have very long asymptomatic contagious periods and very high mortality rates once they become symptomatic". Organisms have evolved to recognize non-self pretty quickly, and pretty good defenses against airborne attacks, which is why 30% is the highest mortality rate for untreated airborne viral illness. Contact viruses can be more deadly, but have limited levels of contagion. I agree that a virus is one most likely route for societal breaking, but compared to the evolutionary space being explored by viruses every day, all the foreseeable AI is a drop in the bucket. When I was in grad school we often said the large numbers would be more accurately described as biological rather than astronomical.
Honestly, I think a lot of people here played too much Pandemic 2, that old Flash game where you could select your bug's traits at will, and then (inexplicably) "update" it as it spread and you gained more XP (and also inexplicably, there was no spontaneous recover/immunity... people had your bug until a vaccine or a drug came along). The key was keep symptoms non-existent until the virus reached Madagascar and New Zealand, then turn on the hemorrhaging and strokes!
Real viruses... do not work like this!
Thanks to Noah for the warning and thanks especially to John Froberg for applying his expertise to this subject.
Good article, Noah, and I completely agree this is the most likely doomsday scenario and not getting enough attention. For those reasons, I would highly recommend you make this a free article so that it can be shared and read more widely.
OK. So it sub-standard and only kills a few million while we work out the vaccine.
6. Since we live in a world with AIs that can create bioweapons the biolab, which is not staffed by angry teenagers who want to release bioweapons, has access to the same AI (probably better) as the angry teenager does and runs the incoming sample requests though the standard "is this a bioweapon" prompt.
Imagine this with human virus designers and human virus checkers. How easy would it be to identify a dangerous utterly novel pathogen from just seeing the genome?
Obviously AI can throw more compute at the problem. But will every biolab do so?
I think you've got the general shape of one piece of the solution right, but there are a lot of obvious scary holes in this safeguard!
Typically in this nature of problem the attacker has the disadvantage because the problem of "design something that will do X" is fundamentally more complex than the inverse "will design X do Y?" This seems similar to "design an airplane that works" vs "look at this airplane design and tell me if it can fly." I'm not an expert in this field, though, so there could be something lurking out there that actually makes the attacker's problem easier than the defender's, but I have never heard an argument for that.
As for the non compliant biolab problem I think that's pretty easy. The screening tools can be publicly funded infrastructure. We can even pay labs a bit of money to send their samples in for screening. This means there is absolutely no incentive not to comply.
The problem is much closer to "order a set of 3D printed parts that can be used to make an airplane" and "predict what this catalog of 500 printed parts assemble into". I think it's very offense dominant.
Fallingknife beat me to the punch. I also think he phrased it too strongly, but the jist of what he wrote is raising a legitimate point. Noah, even underground labs won't want to design killer viruses and whatnot, that's bad for business and for themselves. Have you considered this?
To be clear. I don't think it's impossible for a lab to be so poorly run that it would not conduct a basic AI check before making a a custom made virus, but I'd also expect the correlation between a lab's ability to make a virus and do a (what seems in your hypothetical) a fairly elementary check that can be automated to be pretty negative. If they don't do this, cheap and easy enough that an angry teenager can do it, they're probably not the best and may not be able to make something tricky.
This doesn't eliminate the risk, far from it, and I think you're highlighting this risk correctly. But you may be overestimating the probability here. It may need someone capable of using a lab themselves...
Also. If The Economist is writing articles/leaders on this, it's surely not so far from people's mind at this point.
AIs could probably help with the evasion as well, especially in the future: https://www.rand.org/pubs/research_reports/RRA4741-2.html
Additionally, the assembly could be done in house, and many gene synthesis companies do not screen orders. It's worth noting that bioterror is very rare, and nobody has ever seriously tried with a contagious agent. Thus, in cut-throat low-margin businesses like DNA synthesis services, it makes sense to cut corners here.
Really, what would be the odds that YOUR company would fulfill an order to take out half the world before 2028, given the historical evidence is so weak? I don't agree with this view, obviously, but it's not crazy.
I'm not saying it's impossible, sure, but the hypothetical here is an AI that a not-genius-level teenager is able to use to design a few superviruses successfully. Let's add the necessary premises that I think are needed for this to be plausible, feel free to disagree here.
- Jailbroken, super powerful AIs are not too hard to acquire. (The skill level for a teen to do it would surely match that of a lab)
- These AIs are not very expensive to run. (Teens typically don't have huge resources, ditto the labs)
- These AIs should be pretty good to both design and figure out what a virus does. (It goes without saying, trying to hide characteristics would seem tricky for another AI to detect)
- Underground labs have some significant incentives to not create things that draw attention to themselves and/or kill themselves. (How much is debatable, of course, but the incentives are there)
It only takes one slip up, one badly run but still capable of making superviruses lab, of course. But it probably lowers the risk, to some extent.
So number 1 is true now and will remain true as long as open-weight models keep getting deployed.
Number 2 is probably true. As someone else mentioned, the main cost would be reagents. We're probably talking about a budget of 50-100k, of which a few thousand at most must go to compute. I think a teenager is a bit of an exaggeration here, but not outside of the realm of the possibility.
Number 3 is not needed (for the most part) on the enhancement side because of previously published information. On the obfuscation side, it's likely offense dominant. AIs do have to become better to do this reliably, but the situation laid out in the original post is achievable without superintelligence.
Number 4 probably won't involve underground labs. They'll be legitimately fulfilling orders for pharma companies and small research labs. Some labs will figure out that if you don't screen any orders, you can shave off a few percentage points from order costs. This is a fairly easy coordination problem to solve, but we are not track to solve it due to the low salience and speculative nature of the risk.
To keep things brief. If your assessment of 2 is correct, that does change the calculus further, but decreases the probability a bit more, in my view. Still too high to feel safe, but lower.
3 Your point about the offence might be true, but it seems a bit speculative to me, I'm honestly not sure... and yes, agreed, I'm not discussing a superinteligence scenario here, sorry if my writing (which was a bit hasty) gave that impression.
4 I'm not sure. A few percentage points would be quite huge! But let's put it like this. If I understood you correctly (apologies if not) and the reagent costs are 50-100k, taking that as the baseline cost for manufacturing the viruses, then you're saying that the cost of using AI to screen would be at least 500-1k (that's 1% of the total cost). Doesn't that seem too large? If it's less than 1%, we might still have issues (depending on how competitive the market gets!), but it does seem like a silly thing to skimp out on.
I mean, look. Even then, processes are not perfect and even if I'm right, a lab would need to make one mistake with one rogue agent for everything go wrong... so I'm not doubting the existence of the risk. I'm just feel that this point reduces the probability significantly.
For 4, the providers have to screen everybody, so the cost applies to their total revenue. It really depends on the set up, but to give you a sense of cost, you have to run sequence alignment for every fragment you get and then have a human reviewer verify the flagged ones. I think today, a few percent of gross revenue is accurate, but many of these costs go to research and development of the pipelines. Marginal cost is probably more like a tenth of a percent of the cost of the order. But it could get worse if you insist on stronger methods to beat stronger offense.
Like I said, I think this is a crazy thing to skimp on, but as the industry grows, there will be a lot of competition. Many people just completely dismiss these risks out of hand, so it's likely you'll get a handful of companies that don't screen. Then you just have to identify them with some probes. This applies to both DNA synthesis companies (which exist now and has many examples of non-screeners) and end-to-end cloud labs (which is nascent and could go either way).
"Noah, even underground labs won't want to design killer viruses and whatnot, that's bad for business and for themselves"
What if it's a terrorist biolab?
Or your pissed off teenager is instead a pissed off graduate student at a research university, which means he can find out where the biolab is and has an excuse to be there...
Fair point even if it's not a trivial proposition.
That said, it does raise the question how likely it is for a terrorist organisation to want to create a unlimited reach bioweapon. I don't think most terrorists want to see the total human destruction, unlike the more nihilistic viewpoint Noah proposes.
Still, not an impossible scenario, no, and worryingly so.
Yes, I agree that the majority of them don't.
Just like the majority of people don't.
But I definitely remember different versions of suicide cults over the years.
And of course, the calculus would change if they thought they had some type of vaccine for it
Well, except for the military labs designing bioweapons….and vaccines for their own people. Though the window will be small.
That is maybe a much greater risk than the random teenager and “unlicensed” biolab.
This is easy to get around. Order pieces of DNA that look harmless in isolation then stitch them together yourself into something dangerous. Which not everyone can do, but some people can.
Can you make this a free post so that we can share it widely, given the potential concern you express?
I will, yeah.
Sadly, I must agree with Noah on this one because it's not only the angry teenager but also rogue states (North Korea) and a long list of terrorist organizations that may make this effort. That the downside risk is mass death or civilizational collapse means even a tiny probability of this event must be taken seriously. Given the track record of the human race I've decided to front load my retirement "bucket list". In my 69 years I've never experienced this level of uncertainty about not only my future but all our futures.
Dude, I’m reading “The Stand” right now. Ugh.
Thank you for this column, Mr. Sunshine.
A version of this occurred to me many years ago as an explanation for the Fermi Paradox: That intelligent, technology-oriented life forms only survive until they achieve a level of technology in which any ordinary individual in their society has access to technology capable of wiping out the species. With large enough population size simple variability in individual behaviour dictates that individuals will periodically attempt world destruction--I have seen limited efforts in this direction in the news multiple times throughout my life: Jonestown mass suicide, the Japanese attack you mention, the Manson "family", and more recently Putin threatening suicidal nuclear WWIII if he isn't allowed to crush Ukraine, etc. Once that level of destructive capability exists, it's just a matter of time until someone succeeds. If we look at the destructive capacity of an ordinary individual human, it has been increasing at an accelerating rate for some time. I don't think we are there yet, but I agree that the reasons I see even--especially--experts presenting to dispel alarm actually just increase my alarm because they always seem based on the level of capabilities we have today, while ignoring the accelerating rate of technological development. I'm 74. The things I can do today from my computer, or even just using my phone, would astound my 24 year old self. One of two things will happen, and probably within the next decade: We will either figure out a way to modify or constrain human behaviour, or human behaviour will destroy us all once that ordinary but disturbed individual has access to the capability to wreck that level of destruction.
I come to the same conclusion. We will soon be able to annihilate the human race from the actions of a few bad actors. This is true with or without AI. It is virtually inevitable.
Thus our ONLY hope is that AI can prevent us from doing so. We need a new higher level of intelligence and morality than what the current naked apes are capable of alone.
What struck me is the weakest link: some humans are evil. A line from "Hi Ren" starts playing in my head:
My name it is stitched to your lips, so, you see
I won't bow to the will of a mortal, feeble and normal
You wanna kill me? I'm eternal, immortal
I live in every decision that catalysed chaos
That causes division
I live inside death, the beginning of ends
I am you, you are me, I am you, Ren
I'm dismayed at the senseless destruction caused by wars that seem to arise from human greed: the lust for power, money, land. Gaza, Ukraine, Uganda, WWII, ... If we cannot defeat the immortal forces of evil, then Fermi was right.
I agree. If only these youngsters could see through our eyes.
The extreme recklessness of teenagers is what makes it plausible that he would want to do this, but other things about teenagers militate against him succeeding: there's lots of stuff teenagers don't know, they tend to lose interest in things fairly quickly and have little or no experience at negotiating their way through complex processes. This would be nowhere as easy as breaking into your Dad's gun cabinet. His plan would very probably come off the rails at some point in the following parts of the process:
- finds an unlicensed biolab in East Europe - these are probably not as common as we fear, plus, searching for them online, even via AI, is likely to raise some sort of red flag somewhere in the surveillance community.
- that ships genetically modified viruses - i.e. a supplier that has little interest in their own survival which is very rare
- It . . . makes a potential doomsday virus - the teenager would probably have to have a lot of money (for a teenager) to make this happen and there might be no such thing as a doomsday virus or at least the gate through which any such virus has to pass might be very narrow relative to the design space that has to be explored
- ships these to the angry teenager - again, they would have to have little interest in their own survival
- The teenager mixes the samples together - i.e. without inadvertently destroying them, reducing their potency etc. Rogue biolabs are notoriously remiss when it comes to providing instruction manuals and customer support.
There will be any number of very mundane things that can go wrong for the teenager at each of these points. So getting through each of them alone will be improbable, and each of these low probabilities are fairly independent of each other. But in order to succeed, the teenager has to have all of them happen. That's a lot of low numbers multiplied by one another. I'm not saying it couldn't happen - almost anything could happen - I just think it's extremely unlikely.
It's at these rather workaday points in the process that regulators and governments need to position their roadblocks, to make what is already difficult, well-nigh impossible.
Remember, the teenager himself does basically none of this. The AI agent does this all for him. It doesn't get tired or bored.
Yes, but these objections aren't about efficiency in search and information processing or effectiveness in problem-solving - the kinds of thing that AI can do. They are about real-world probabilities (e.g. rogue, affordable E. European biolabs being rare) and hands-on expertise: the sort of things that you can only learn by practice, seeing and doing repeatedly. Even at undergrad level chemistry students spend a lot of their time in the lab, learning their craft. There would probably also be implicit knowledge barriers to finding, contacting and ordering from the rogue labs: criminal enterprises don't tend to advertise themselves as such and often prefer word-of mouth referral and face-to-face communication to avoid being detected or leaving an evidence trail.
AI still has a long way to go before it can replicate and supply these types of implicit knowledge. And getting this type of knowledge would probably require AI to be embedded in physical devices which are far easier to track and regulate.
You are being too specific. NS outlined one possibility, reckless teenager. Here's another: bitter suicidal biologist at sketchy biolab (doesn't need to be criminal lab, just sloppy and lazily administered).
Maybe this case is also a matter of multiplying small numbers, but the number of cases might be large . Multiply small numbers enough times and you get a positive.
Psycho biologist in small nasty nation's newly feasible program. Same nation's biolab program but the antidote is faulty. Same scenario but the antidote works and the dictator is crazy enough to act. I could go on.
The main point is capability. If AI makes work that previously required large coordinated teams feasible for an individual you've got a problem. Intelligent psychopaths exist, the AI itself doesn't need to turn on us.
Would the psycho biologist need AI? My point is that with people who are clever but not mature, experienced or socialised enough to have any accountability whatsoever (and so are prepared to take the whole world with them), you start with very low numbers which then get thinned out rapidly by hurdles, which they face because they're working alone and are insufficiently worldly to know how to deal with them, each of which stops all but a very small %. If the point is that a rogue state or well-organised death-cult-style terrorist group can with AI now, for the first time, manufacture biological weapons that are so deadly they could wipe out the whole world as we know it - that scenario runs into the objection that there are staggeringly few such groups or people. Not wanting there to be a world anymore and being able to do what's necessary to bring that about, even with the assistance of AI, are in serious tension. The overlap between the two is very small, potentially non-existent. It's certainly a risk, but in pretty much any telling of it that I can imagine, it requires so many moving parts that success is incredibly unlikely.
Yes, he would need AI.
The current leading edge science is team work. An extremely gifted biologist might create one virus, but not 100. This involves different steps (design, prototype, multiplication, introduction to the wild) that would likely be separate specialties. All those "hurdles caused by their being alone" are quite easily overcome with a sufficiently capable AI.
It is the current paradigm that you need teams for everything that matters and assembling a functional team of unstable psychopaths is nearly impossible.
When AI empowers the individual, that safety is gone.
See my response to Noah above. Most of the hurdles created by being alone are not easily overcome because they're not a matter of having more hands but of having hands that have acquired implicit skills and judgement passed down from previous practitioners and that are connected to heads with contextual knowledge and bodies that can win the trust of bad actors with highly suspicious natures. AI will, at the very least, take time to duplicate these and may never be able to, at least not in a way that regulators wouldn't be able to keep pace with.
The recent robot Olympics in China provided a glimpse into this issue. In the races, the robots were very good at starting and running but slowing down whilst going round a bend in the track and avoiding other runners - something that human sprinters do with effortless ease - is immensely complex. So their creators didn't even bother trying to make them do it. They just let them run into crash mats, often wrecking themselves in the process.
I think that for the foreseeable future our teenager's plan for world destruction would also probably end up amongst the crash mats for want of a range of implicit skills that AI can't mimic.
As long as you picture a teenager, I tend to agree.
But once you have a disgruntled scientist in mind, things change. A single person with good judgment and implicit skills combined with the leverage of AI could wreak incredible havoc.
Yes, that's true. But then such a person could do a fair bit of damage before the advent of AI. Now maybe they could do more or do it a bit more easily or there could be more people who could stretch their expertise across into areas of threat than there were previously. But it's not the transformation of the risk landscape that it would be if a smart teenager could pull it off. And some of the barriers highlighted would remain even for a skilled, experienced scientist.
Seems to me the problem is the biolab.
Anyone consider it strange that stricter regulation, transparency and global coordination over biolabs wasn’t an outcome of Covid?
Almost like the grifters set the narrative.
Wuhan lab still open for business?
It's quite possible that we now have stricter regulation, better coordination, and more transparency *among virologists and their regulators*, but done on the quiet by the insiders in virology research. But nobody on the inside has an incentive to be more transparent *to the general public*.
You need intelligence agencies to flood both sides of the mail order custom pandemic market with honey traps. Today, if you try to find a community online that will help you be the next McVeigh, you will very likely wind up talking to the Feds.
Should be the same for would-be bioweapons. There should be ubiquitous crowds of fake buyers that trigger a visit from agents or drones to anyone who accepts their order. And fake unlicensed labs that trigger the same to any would-be buyers. Search is a hard problem, and fortunately it’s easy to find out where the AIs will look first.
There might well be already. In regard to nuclear weapons, I've read that it's quite legal to buy krytons, which are needed to trigger nuclear explosions, though trying to do so will get you a lot of attention from the feds.
I’m sure the feds monitor a great many things that would be useful in nuclear weapons production, but the truth is a basic nuclear bomb isn’t complicated or hard to make. I learned the science behind it in second year of a physics degree, and after that you need the materials and some reasonable precision engineering.
The hard part is getting the enriched uranium or plutonium. I’d expect that the fissile materials and things like centrifuges for making them are what are really heavily monitored.
You do have to get good enough at both alignment and international coordination that your fake buyers and fake sellers don’t wind up trying to arrest or drone strike each other, which is non-trivial.
Why Nirvana? Why Heathers? Why not Phillip Glass or Taylor Swift? Ok. Good article as usual. Scary stuff truly.
Nirvana because it makes you depressed and angry to listen to.
Heathers because J.D. thinks humanity sucks and wants to kill people out of pique.
Glad you liked the article. ;-)
How exactly are the models going to improve to the point where they can do this? The reason AI has improved so dramatically since 2023 is largely due to quotidian software engineering around harnesses, not breakthroughs in model design. The models themselves are still transformer-based models, and the improvement has largely been in scope rather than in kind—i.e. bigger context windows, better harnesses, and efficiencies at inference.
Claude Code is magical for computer code because the internet provides extremely robust training data for extant coding languages. But, as the molecular biologist mentioned, relevant open-source biological data is much sparser and spottier. The operation being imagined here would also be more or less unprecedented in human history, so the data on how to actually execute such a plan is sparse as well. An LLM asked to fill in those gaps would likely default to fairly crude works of fiction.
This matters because I am dubious that skill in hacking will simply extend to skill in bioterrorism. There is a lot of data on how to hack successfully and not much on how to do bioterrorism—or, for that matter, how to do large swathes of work in the biomedical field. The analogy between an AI becoming an extraordinarily capable programmer and becoming an extraordinarily capable biologist therefore seems much weaker than the article assumes.
And the problem is not merely figuring out how to do something unprecedented. It would also need to figure out how to evade detection in the process. Details on how specific surveillance systems work are themselves unlikely to be comprehensively represented in easily accessible training data. The model would therefore need to use its imagination both to figure out how to actually execute the plan and to anticipate systems and circumstances for which it has little or no precedent. Modern LLMs are not reliably capable of doing that. I have never once encountered a useful hallucination. In fact, it feels much more likely that a layman doing something so complicated would end up in a blind alley after the model took a hallucination and sprinted off the reservation with it.
If we assume the progress in generative AI continues as it has since GPT-3.5 into 2029, the villain here will be using a model which still has all these shortcomings. Throwing more compute, making context windows larger, making tokens cheaper etc will only be diminishing returns if the scaling curve is sigmoidal rather than exponential, and there is good reason to believe that it is, and has been for ~18 months now.
What we’re talking about is an AGI capability here, and I don’t think the current approach of LLMs in harnesses are AGI. To get there, the labs are clearly going to need another breakthrough on par with Attention is All you Need. I won’t profess to know what breakthrough gets us to AGI—Yann LeCun says it’s world models, which is as good a hypothesis as any. Sam and Dario hand wave this problem away by saying their models will magically find this breakthrough on its own. This is again something that would require an imagination their systems are structurally incapable of.
So the part of the 2029 scenario I find unconvincing is not the premise that AI capabilities will continue improving. It is the assumption that continued improvements of the kind we have seen since GPT-3.5 necessarily produce the qualitatively different capabilities this scenario requires. The doomsday-virus argument quietly assumes that scaling and better agentic scaffolding will somehow turn a system that is exceptionally good at recombining abundant precedent into one that can reliably solve a sparse-data, unprecedented, long-horizon problem in the physical world. That is precisely the leap that needs to be demonstrated rather than assumed.
"quotidian software engineering around harnesses" is not true.
Post training for in harness agentic behaviour is the key. You can test it right now: put an early model in a harness and try and build something with it. It won't be pretty.
Without getting hung up on what is quotidian, I don’t think the way they’ve improved post-training is a fundamental breakthrough rather than an in-kind improvement on their existing methodologies. Which is to say they are iterating on the software pipelines they’ve built to crank out these models. More to my point, the improvement in agentic systems is because these companies are using the biggest pots of money ever to specifically be good at using these harnesses for software projects. I don’t believe they are doing so for biology, and if they are doing so, they absolutely should not be.
I agree with this.
A lot of people are working on real learning and real understanding. One approach is Accelerated Understanding Inc.
You are correct, that you need more than just a LLM with sufficient scale. But building this "more" is exactly the current frontier.
Since current AI makes experimentation and iteration vastly cheaper and easier than in the old days and really bright minds provide the direction and philosophy, I don't expect it to take very long.
Two quibbles: You are vastly underestimating the damage a nuclear war would cause, though it would certainly solve the global warming problem for a while. Estimates are that 99% of those living in the northern hemisphere would die, the survivors from starvation. There may be fewer bombs then the Cold War days but they are much much more destructive.
And in the bioterror scenario, with much less physical destruction, I suspect a survival rate of 10 percent or 800-900 million people with access still to libraries could manage to live significantly better than a Neolithic lifestyle.
How are modern bombs more deadly than earlier nukes?
1. Most bomb designs are old.
2. As missile guidance got better the bombs got smaller, no need to overcome a large blast radius with megatons.
The most destructive bomb ever? 50 megatonTsar Bomba, 1961. And it was de-tuned, the U-238 casing removed. Unaltered it was 100 Megaton.
I don't know if NS is right about survival, but average nuke explosive power has trended down, not up.
Smaller bombs are not less destructive: "multiple smaller weapons are more effective than a single larger one. Twenty 50-kiloton warheads, for example, destroy nearly three times the area leveled by a numerically equivalent 1-megaton weapon." MIT report
Don’t forget the EMP effect of nukes. A nuclear war is likely to open with a volley that includes using EMP nukes in orbit to wipe out satellites en masse. That would also knock out electronics on the ground in a massive radius.
Also, I agree with tomtom below. By all accounts, the known nuclear stockpiles are smaller warheads than early to mid Cold War because many smaller warheads are more effective than one big one (due to the inverse square law of energy dissipation).
No one can regulate what they don’t understand. Especially the caliber of men we blithely elect to run our country. This has happened over and over in our history with technology, science and invention. Private corporations end up doing the unregulated research and introducing products to make profits often with potentially stunning consequences. Take forever chemicals and genetically modified crops for example. I remember 60 years ago when effective genetic manipulation techniques were discovered and went unregulated for years. The same is happening now with AI. We have always lived with this problem like hopeful, trusting lemmings, and one day it could easily destroy us. It won’t be man’s intelligence that kills him but his ignorance and fear of what and whom he does not understand.
ARPA-H seems to be working on a plausible mitigation effort: an indoor air-monitoring program that aims to create "novel biosensors and risk assessment technologies [that] are designed to detect and respond to airborne biological threats earlier than ever before and to build a future where healthier buildings lead to healthier lives."
https://arpa-h.gov/news-and-events/how-arpa-h-revolutionizing-indoor-air-quality-and-why-it-cant-wait
A small wrinkle to add to your scenario: even if a vaccine is created, some people might refuse to take it due to the effects of vaccine misinformation from the Covid era. (In my opinion, that is going to make any future pandemic worse, and thus make pandemics a more effective way of killing people).
By the way, recommend Station Eleven as interesting on how a population devastating virus might play out.