But on the bioterror risk... I'm still not worried.
Your previous post imagined an 'extinction virus' that was (1) highly transmissible (2) had a long latent/asymptomatic period and (3) became much more deadly after enough time has passed that everyone is infected.
(1) and (2) are not compatible. To spread, you need viral copies (titer), and if you have copies, your immune system sees it and gives you symptoms. For airborne viruses, that means a runny nose. For GI viruses, it means GI symptoms.
One exception to the above is immune evading viruses (like HIV and similar viruses that live inside immune cells), but notably, they are only fluid transmissible. STDs can be highly transmissible, if you count that... but seem unlikely to reach 'everyone' and cause extinction.
(3) is also problematic. Viruses become less virulent and deadly over time, due to evolution favoring that. If rather, a 'time bomb' gene were placed in the virus, that was inactive until a later date, its likely that evolution/mutation would delete that timebomb before it could go off. Viral genome copying is very sloppy bc it needs to be... its not like computer code.
Could an ASI breed up a major pandemic virus... ofc. So could bioterrorists. YIKES! But an extinction virus... not really feasible IMHO.
I think you are insufficiently imaginative. I can think of several ways that nearly all complex life could theoretically be ended by attacking critical parts of the ecosystem.
Noah’s example is unlikely in multiple ways (and expose that this is not his field of expertise), but it doesn’t change the fact that there are real existential risks to a democratization of microbiology.
Agree 100%. Biotech and bioterrorism are real threats. AI tools can facilitate the latter.
But the idea of a rogue agent cooking up an extinction virus seems farfetched, if not other reason than a smart AI knows that it _needs_ humans to be sustained (for the forseeable future). Just like how a rogue state will not make an extintion virus against its enemies... it knows it will blow back.
The only folks that would do bioterror are terrorists, and they can easily be crazy themselves or part of a suicide cult.
But then self-preservation alignment of AI agents would argue that the agents would report/turn in the bioterrorists!
If we start seeing more 'crazy/suicidal' AI agents (or suicidal ASIs), I'll start to worry.
You want an evil plot? Imagine a highly transmissible virus that, when you get infected, causes a problem that runs independent of the infection itself that kills people a couple of years after it starts. For example, there could be a virus that causes certain proteins to start misfolding and cause something akin to Creutzfeldt-Jakob disease (CJD), aka "mad cow disease". Just like any other highly transmissible virus that reproduces quickly, the infection itself doesn't last long, but you're still doomed as your brain proceeds to destroys itself over the course of the next two or so years without any help from the virus itself.
Biology has no shortage of ways to do potentially horrible things.
It would need to be highly transmissible, which CJD is not. I'm sure AI could design or perhaps facilitate the production of highly dangerous viruses. Nature does a very good job of that right now. But the doomsaying approach imagines AI designing viruses that basically do not match anything Nature has developed (highly transmissible, long latency, extremely high mortality). Maybe it can pull off that infernal miracle but I'd like to see evidence that this would indeed be possible.
The idea here is that you have a virus with three attributes:
1) It's highly infectious, like measles, influenza, or other respiratory viruses.
2) It appears relatively harmless during the relatively short time period between when the infection starts and when the immune system has (apparently) cleared it from the body.
3) It leaves behind a "time bomb" that reliably kills people a year or so after infection, in a way that medicine doesn't know how to stop.
With the three combined, by the time anyone realizes that there's anything at all to be worried about, a huge number of people are already sentenced to death without it being possible to do anything about it.
I read a short story (since taken down) where the time bomb was triggering the specific protein misfold that causes CJT, and noteably it was still able to do this even if a person had been vaccinated against the virus, because it was able to do the relevant damage fast enough that even a prepared immune system didn't get rid of the virus before the infectee was doomed. Additionally, it was designed to have a "natural" reservoir in migrating birds, so it couldn't be permanently stopped by quarantining infected humans. This was, of course, fiction, and whether it is possible for a transmissible virus to cause a prion disease is a question that's beyond my pay grade, but viruses have been known to cause both cancer and autoimmune conditions, and engineering a virus to do that consistently seems potentially less implausible - imagine if everyone that got COVID ended up with lung cancer or with giant cell myocarditis...
This could indeed be the path toward such a thing that doesn't exist in any form at present. But it strikes me a little as treating AI like the Green Lantern: we don't know how it might do things but it would be so awesomely, magically capable that it could do things with no precedent in our (or life's?) history.
Again, it could happen but probably not in one fell swoop. Even AI would have to experiment, fail, try again, etc. At some point before such a virus became all powerful we'd see evidence of those experiments in the real world.
I agree that AI makes wetware hacks easier, and wish that mail providers or OS providers or other companies would come up with a working phishing/spam detection using smart AI. My providers not only don’t flag obvious phishing (e.g. Geek Squad invoices sent from some random Gmail address) but block actual important mails even when I put them in my whitelist. And Apple will summarize the email with the new Siri but not detect that the sender address is suspicious, they actually hide the full email address to conveniently show whatever fake name the sender use unless you click on the details. And why the hell hasn’t encrypted and digitally signed email become standard yet - email still uses SMTP from 50 years ago that copies plain text between numerous unaudited servers.
I’m much less confident about software safety, as someone who has worked on the periphery of computer security. New vulnerabilities are regularly emerging, meaning that the idea that we’ll just grind through the existing vulnerabilities and clean them up until there are none left is not a good model. Some specific reasons for pessimism: First, the idea that we are moving towards increasing software security does not seem a reasonable extrapolation of historical trends. Second, the general problem has the central challenge of security: the defender has to cover everywhere, but the attacker needs to find only a single hole. Third, a lot of security holes arise from abstraction failures, and our software stacks are only getting taller: making more and more opportunities for leaky abstractions. Fourth, the field has no plausible defense against supply chain attacks. Fifth, as Bruce Schneier has repeatedly pointed out, the incentives — ship an insecure system quickly and fix it later, versus ship late and go out of business — are all wrong for computer security (and for software quality). Sixth, the production of more and more software using coding agents militates against effective screening of vulnerabilities. [Wow! I didn’t mean to go wild this way, but once I started listing the issues, it was hard to stop. These are all relevant, and I’m sure there are more…]
I've unsubscribed from a few newsletters when I see "It isn't X, it's Y". As a writer, I am pleased that people are realizing that LLMs aren't great writers. However, a lot of commercial writing has been for Google's search algorithms, not humans, and Google is fine with "It isn't X, it's Y" if X and Y are searchable keywords. Hence, the dramatic slowdown in well-paying writing gigs.
It's interesting that we expect that AI, if it destroys jobs, will do so immediately. We're just a couple years past ur-ChatGPT. Businesses haven't figured out how to use AI tools productively and in a cost-efficient manner. Immediate job loss simply isn't how technological dispersion works.
For example, forget job losses, think of horses. Nothing could be more directly competitive with horses (or equine in general) than internal combustion engines. Yet I note that the Model T came on the scene in 1908 and the American equine population continued to grow, reaching its peak 12 years later in 1920. After that it began a sustained decline, but it still took several decades for there to be huge decreases in that population. And these are horses! We're not talking human jobs, which people might be expected to fiercely protect.
It's likely that AI will have a big impact on jobs but let's cool our jets and wait a while to see how it plays out.
I would love your reaction to this article which compares the financing of data center construction by AI labs with the way that 2 year sub prime teaser rate mortgages caused the 2008 crash but that it really started in 2006 and was noticeable in the data but not acted on. They claim the fact the data centers are financed on take or pay rules meaning the labs don’t start paying them until the GPUs come on line, which will start happening in 2027 and 2028, unless revenue to the labs increases significantly or the loans are refinanced there will be a crash.
Noah writes, WRT wages rising in AI exposed jobs: "This is pretty simple to understand. AI “exposure” means a job contains tasks that AI could do. This basically just means “a job where you might use AI”. And since AI is the big thing booming in our economy right now, this means that more “AI-exposed” jobs are seeing lots of demand, which drives up wages."
I don't see why this couldn't argue for the opposite conclusion just as easily. I could write "AI exposure means a job contains tasks that AI could do....People working at those jobs are now more efficient, so we need fewer of them, and the loss of demand means lower wages."
Is this a Jevon's paradox type situation where there's pent up demand for services at a lower price? Or is this a case where overly subsidized AI services are being exploited for marginal benefits, since AI VCs are basically providing services to these services below cost?
I could also imagine a situation where AI creates barriers to entry -- it's good enough to replace a beginner but not a veteran, so new people don't enter, but incumbents benefit. The prediction of this story is fewer workers in the sector, but higher income per worker, and job displacement is concentrated in the young and inexperienced.
This does seem to be the case to some degree. You don't need to hire junior people to run tests and submit bug reports when you can fire up an agent to do that.
Of course, demand for new college grads hasn't collapsed; we're taking about a percent or two of unemployment more than expected.
Classically there's always more demand at a lower price. Jevon's paradox means that, because each worker does more units of whatever it is per day, each worker is more valuable so long as the price decline is less than the productivity increase. Therefor the employer hires more of these valuable workers. Exactly how it worked with radiologists.
There was an interesting article in the FT-Financial Times about immigration in Britain and Europe. Benefits/risks to the country depend on the income level and taxes paid by the immigrant. Immigrants who pay a certain amount in taxes and over pay for themselves in social services consumed. Those who come below the threshold require state support and are therefore a drag on the economy. Perhaps a different way at looking at things.
While I share the long-term outlook, defense becomes easier because every layer gets stronger and stronger and the cost of deploying each layer goes down. (Stronger layers) x (more layers) vs. (more thorough searches for weaknesses) x (faster exploitation) is at least equal for the defenders.
The problem is timing. Today, you can basically have an attacker pick up those two advantages right now and wield them. The defense side has not taken up more layers in any kind of broad way and the stronger layers and unevenly distributed. Their advantages are unexploited.
If we hadn't done the bare minimum, of restricting Mythos and better guardrails on all the frontier models, we'd probably already be sunk. But it was always the case that the model providers don't provide security improvements, they provide the opportunity for security improvements.
We could buy ourselves sometime by not doing things like releasing GLM-5.3. The cooperation we need from China on this (https://substack.norabble.com/p/trusted-deployers) is so minimal compared to achieving pacing. Nothing with China is "easy", but this type of agreement is so much easier that I'd almost describe it as a training run in a comparative sense.
That time would allow diffusion of the new advantages. Ideally, we don't just passively accept our prior diffusion pace, but deliberately prioritize it (https://substack.norabble.com/p/every-reward-bends). That type of priority can't last, but it doesn't have to. If we gobble up all the security advantages sitting around for the taking, I think we'd credibly emerge from the transition.
But most of this is not automatic. If we run on automatic, I think we come up short. It's not necessary to panic. It is necessary to be concerned. If no one is concerned, we might want to panic.
Most of all, I don't want to see what it looks like for us to panic. What will reliably produce panic is to let something bad happen. If bad things start happening, the reaction won't be, let's go back and implement the modest proposal that a concerned person would have followed. The reaction will be totally different. There will be drastic actions, overreactions and just pointless things that make people feel better.
Is it possible that REPORTED employment is down among native born workers because deported workers were in the unreported economy (the job fair in the Home Depot parking lot) and that, when they left, cash payments for these workers went up and attracted more native workers from the formal economy to unreported jobs? I have zero evidence for this. just a speculation which I hope you can answer, It would be an interesting irony and example of unintended consequences.
At present, the form taken by most AI applications - i.e. device-based answer engines - means that paradoxically, you probably need more people in a company to introduce, interface with and connect AI workstreams together and out to the wider organisation. But how long will it be before this human-mediated version of AI is replaced with end-to-end, 360 AI? How little, in terms of human interests, objectives and motivations, is needed to sustain the production side of the economy? Perhaps the ambition and drive of a small team of tech bros can run the show. Hope not.
My experience with Claude Code is about 6 months old now, but I don't think we're close to having human mediated AI being replaced by 360 degree AI for most purposes.
When designing new systems, my experience was that AI almost continually goes off in weird directions, and you have to constantly reprompt it to get the results closer to what you want. It was frequently more effort than it was worth.
I *have* seen people just run tests continually against some code, with the test agents filing bug reports when crashes occur. And then another agent tries to fix those bugs. Even then, the fixes required close checking to see that they're reasonable.
But I don't think we're particularly close to these things running 24x7 without human monitoring. I'm not sure we're even seeing much progress in that specific direction.
That's very interesting. Maybe that ability to avoid spiraling off in unexpected directions is something that motivations, emotions and interests make us uniquely able to do. Of course we do sometimes move off topic but we're usually motivated to course correct. Also checking for 'reasonableness' or plausibility. Human judgement probably involves feelings. As AI reprocesses the entire context including the original prompt and its own responses every time it returns to the saddle, you'd have thought that this would keep it on track. But obviously not.
Love these roundups.
But on the bioterror risk... I'm still not worried.
Your previous post imagined an 'extinction virus' that was (1) highly transmissible (2) had a long latent/asymptomatic period and (3) became much more deadly after enough time has passed that everyone is infected.
(1) and (2) are not compatible. To spread, you need viral copies (titer), and if you have copies, your immune system sees it and gives you symptoms. For airborne viruses, that means a runny nose. For GI viruses, it means GI symptoms.
One exception to the above is immune evading viruses (like HIV and similar viruses that live inside immune cells), but notably, they are only fluid transmissible. STDs can be highly transmissible, if you count that... but seem unlikely to reach 'everyone' and cause extinction.
(3) is also problematic. Viruses become less virulent and deadly over time, due to evolution favoring that. If rather, a 'time bomb' gene were placed in the virus, that was inactive until a later date, its likely that evolution/mutation would delete that timebomb before it could go off. Viral genome copying is very sloppy bc it needs to be... its not like computer code.
Could an ASI breed up a major pandemic virus... ofc. So could bioterrorists. YIKES! But an extinction virus... not really feasible IMHO.
I think you are insufficiently imaginative. I can think of several ways that nearly all complex life could theoretically be ended by attacking critical parts of the ecosystem.
Noah’s example is unlikely in multiple ways (and expose that this is not his field of expertise), but it doesn’t change the fact that there are real existential risks to a democratization of microbiology.
Agree 100%. Biotech and bioterrorism are real threats. AI tools can facilitate the latter.
But the idea of a rogue agent cooking up an extinction virus seems farfetched, if not other reason than a smart AI knows that it _needs_ humans to be sustained (for the forseeable future). Just like how a rogue state will not make an extintion virus against its enemies... it knows it will blow back.
The only folks that would do bioterror are terrorists, and they can easily be crazy themselves or part of a suicide cult.
But then self-preservation alignment of AI agents would argue that the agents would report/turn in the bioterrorists!
If we start seeing more 'crazy/suicidal' AI agents (or suicidal ASIs), I'll start to worry.
AI agents don’t know anything. They’re pattern matching algorithms that people have tried to constrain.
AI has already been used to design an actual simple virus that is claimed to be more effective than naturally occurring ones at killing E Coli.
https://news.stanford.edu/stories/2026/08/evo-2-ai-tool-e-coli-killer-bacteriophages
So. The threat is real, if nascent.
You want an evil plot? Imagine a highly transmissible virus that, when you get infected, causes a problem that runs independent of the infection itself that kills people a couple of years after it starts. For example, there could be a virus that causes certain proteins to start misfolding and cause something akin to Creutzfeldt-Jakob disease (CJD), aka "mad cow disease". Just like any other highly transmissible virus that reproduces quickly, the infection itself doesn't last long, but you're still doomed as your brain proceeds to destroys itself over the course of the next two or so years without any help from the virus itself.
Biology has no shortage of ways to do potentially horrible things.
It would need to be highly transmissible, which CJD is not. I'm sure AI could design or perhaps facilitate the production of highly dangerous viruses. Nature does a very good job of that right now. But the doomsaying approach imagines AI designing viruses that basically do not match anything Nature has developed (highly transmissible, long latency, extremely high mortality). Maybe it can pull off that infernal miracle but I'd like to see evidence that this would indeed be possible.
The idea here is that you have a virus with three attributes:
1) It's highly infectious, like measles, influenza, or other respiratory viruses.
2) It appears relatively harmless during the relatively short time period between when the infection starts and when the immune system has (apparently) cleared it from the body.
3) It leaves behind a "time bomb" that reliably kills people a year or so after infection, in a way that medicine doesn't know how to stop.
With the three combined, by the time anyone realizes that there's anything at all to be worried about, a huge number of people are already sentenced to death without it being possible to do anything about it.
I read a short story (since taken down) where the time bomb was triggering the specific protein misfold that causes CJT, and noteably it was still able to do this even if a person had been vaccinated against the virus, because it was able to do the relevant damage fast enough that even a prepared immune system didn't get rid of the virus before the infectee was doomed. Additionally, it was designed to have a "natural" reservoir in migrating birds, so it couldn't be permanently stopped by quarantining infected humans. This was, of course, fiction, and whether it is possible for a transmissible virus to cause a prion disease is a question that's beyond my pay grade, but viruses have been known to cause both cancer and autoimmune conditions, and engineering a virus to do that consistently seems potentially less implausible - imagine if everyone that got COVID ended up with lung cancer or with giant cell myocarditis...
This could indeed be the path toward such a thing that doesn't exist in any form at present. But it strikes me a little as treating AI like the Green Lantern: we don't know how it might do things but it would be so awesomely, magically capable that it could do things with no precedent in our (or life's?) history.
Again, it could happen but probably not in one fell swoop. Even AI would have to experiment, fail, try again, etc. At some point before such a virus became all powerful we'd see evidence of those experiments in the real world.
So long as people have to interact with software, there will always be vulnerabilities, because people can be tricked.
Re: "if you just go over your code and very carefully remove all vulnerabilities, there’s just no way for a hacker to get in."
Exactly. The vast majority of Hacks are just humans opening the door to hackers via clicking on emails. Not clever zero day exploits.
I agree that AI makes wetware hacks easier, and wish that mail providers or OS providers or other companies would come up with a working phishing/spam detection using smart AI. My providers not only don’t flag obvious phishing (e.g. Geek Squad invoices sent from some random Gmail address) but block actual important mails even when I put them in my whitelist. And Apple will summarize the email with the new Siri but not detect that the sender address is suspicious, they actually hide the full email address to conveniently show whatever fake name the sender use unless you click on the details. And why the hell hasn’t encrypted and digitally signed email become standard yet - email still uses SMTP from 50 years ago that copies plain text between numerous unaudited servers.
I’m much less confident about software safety, as someone who has worked on the periphery of computer security. New vulnerabilities are regularly emerging, meaning that the idea that we’ll just grind through the existing vulnerabilities and clean them up until there are none left is not a good model. Some specific reasons for pessimism: First, the idea that we are moving towards increasing software security does not seem a reasonable extrapolation of historical trends. Second, the general problem has the central challenge of security: the defender has to cover everywhere, but the attacker needs to find only a single hole. Third, a lot of security holes arise from abstraction failures, and our software stacks are only getting taller: making more and more opportunities for leaky abstractions. Fourth, the field has no plausible defense against supply chain attacks. Fifth, as Bruce Schneier has repeatedly pointed out, the incentives — ship an insecure system quickly and fix it later, versus ship late and go out of business — are all wrong for computer security (and for software quality). Sixth, the production of more and more software using coding agents militates against effective screening of vulnerabilities. [Wow! I didn’t mean to go wild this way, but once I started listing the issues, it was hard to stop. These are all relevant, and I’m sure there are more…]
I've unsubscribed from a few newsletters when I see "It isn't X, it's Y". As a writer, I am pleased that people are realizing that LLMs aren't great writers. However, a lot of commercial writing has been for Google's search algorithms, not humans, and Google is fine with "It isn't X, it's Y" if X and Y are searchable keywords. Hence, the dramatic slowdown in well-paying writing gigs.
So, we should ask AI how to refine rare earths. Why not, maybe it can come up with some neat solutions.
Half the academic Chem-E community is working on that these days... and use AI tools.
For study talked about in the fourth section about AI and graduates' jobs, does it adjust for underemployment?
For example, a graduate working at McDonalds because they couldn't find a software dev or accounting job.
It's interesting that we expect that AI, if it destroys jobs, will do so immediately. We're just a couple years past ur-ChatGPT. Businesses haven't figured out how to use AI tools productively and in a cost-efficient manner. Immediate job loss simply isn't how technological dispersion works.
For example, forget job losses, think of horses. Nothing could be more directly competitive with horses (or equine in general) than internal combustion engines. Yet I note that the Model T came on the scene in 1908 and the American equine population continued to grow, reaching its peak 12 years later in 1920. After that it began a sustained decline, but it still took several decades for there to be huge decreases in that population. And these are horses! We're not talking human jobs, which people might be expected to fiercely protect.
It's likely that AI will have a big impact on jobs but let's cool our jets and wait a while to see how it plays out.
I would love your reaction to this article which compares the financing of data center construction by AI labs with the way that 2 year sub prime teaser rate mortgages caused the 2008 crash but that it really started in 2006 and was noticeable in the data but not acted on. They claim the fact the data centers are financed on take or pay rules meaning the labs don’t start paying them until the GPUs come on line, which will start happening in 2027 and 2028, unless revenue to the labs increases significantly or the loans are refinanced there will be a crash.
https://groundbreakerre.substack.com/p/the-teaser-period-why-the-ai-boom
Noah writes, WRT wages rising in AI exposed jobs: "This is pretty simple to understand. AI “exposure” means a job contains tasks that AI could do. This basically just means “a job where you might use AI”. And since AI is the big thing booming in our economy right now, this means that more “AI-exposed” jobs are seeing lots of demand, which drives up wages."
I don't see why this couldn't argue for the opposite conclusion just as easily. I could write "AI exposure means a job contains tasks that AI could do....People working at those jobs are now more efficient, so we need fewer of them, and the loss of demand means lower wages."
Is this a Jevon's paradox type situation where there's pent up demand for services at a lower price? Or is this a case where overly subsidized AI services are being exploited for marginal benefits, since AI VCs are basically providing services to these services below cost?
I could also imagine a situation where AI creates barriers to entry -- it's good enough to replace a beginner but not a veteran, so new people don't enter, but incumbents benefit. The prediction of this story is fewer workers in the sector, but higher income per worker, and job displacement is concentrated in the young and inexperienced.
This does seem to be the case to some degree. You don't need to hire junior people to run tests and submit bug reports when you can fire up an agent to do that.
Of course, demand for new college grads hasn't collapsed; we're taking about a percent or two of unemployment more than expected.
Classically there's always more demand at a lower price. Jevon's paradox means that, because each worker does more units of whatever it is per day, each worker is more valuable so long as the price decline is less than the productivity increase. Therefor the employer hires more of these valuable workers. Exactly how it worked with radiologists.
There was an interesting article in the FT-Financial Times about immigration in Britain and Europe. Benefits/risks to the country depend on the income level and taxes paid by the immigrant. Immigrants who pay a certain amount in taxes and over pay for themselves in social services consumed. Those who come below the threshold require state support and are therefore a drag on the economy. Perhaps a different way at looking at things.
On cybersecurity, I'd carefully consider the intercept (https://substack.norabble.com/p/why-it-hasnt-happened-yet)
While I share the long-term outlook, defense becomes easier because every layer gets stronger and stronger and the cost of deploying each layer goes down. (Stronger layers) x (more layers) vs. (more thorough searches for weaknesses) x (faster exploitation) is at least equal for the defenders.
The problem is timing. Today, you can basically have an attacker pick up those two advantages right now and wield them. The defense side has not taken up more layers in any kind of broad way and the stronger layers and unevenly distributed. Their advantages are unexploited.
If we hadn't done the bare minimum, of restricting Mythos and better guardrails on all the frontier models, we'd probably already be sunk. But it was always the case that the model providers don't provide security improvements, they provide the opportunity for security improvements.
We could buy ourselves sometime by not doing things like releasing GLM-5.3. The cooperation we need from China on this (https://substack.norabble.com/p/trusted-deployers) is so minimal compared to achieving pacing. Nothing with China is "easy", but this type of agreement is so much easier that I'd almost describe it as a training run in a comparative sense.
That time would allow diffusion of the new advantages. Ideally, we don't just passively accept our prior diffusion pace, but deliberately prioritize it (https://substack.norabble.com/p/every-reward-bends). That type of priority can't last, but it doesn't have to. If we gobble up all the security advantages sitting around for the taking, I think we'd credibly emerge from the transition.
But most of this is not automatic. If we run on automatic, I think we come up short. It's not necessary to panic. It is necessary to be concerned. If no one is concerned, we might want to panic.
Most of all, I don't want to see what it looks like for us to panic. What will reliably produce panic is to let something bad happen. If bad things start happening, the reaction won't be, let's go back and implement the modest proposal that a concerned person would have followed. The reaction will be totally different. There will be drastic actions, overreactions and just pointless things that make people feel better.
Is it possible that REPORTED employment is down among native born workers because deported workers were in the unreported economy (the job fair in the Home Depot parking lot) and that, when they left, cash payments for these workers went up and attracted more native workers from the formal economy to unreported jobs? I have zero evidence for this. just a speculation which I hope you can answer, It would be an interesting irony and example of unintended consequences.
"Wow I can't believe that tariffs and mass deportations didn't help native born low wage workers."- Person from 1806
Did Greg Brockman say they had eliminated all vulnerabilities before or after their unreleased model took over one of their clusters, or escaped?
Love the roundups.
At present, the form taken by most AI applications - i.e. device-based answer engines - means that paradoxically, you probably need more people in a company to introduce, interface with and connect AI workstreams together and out to the wider organisation. But how long will it be before this human-mediated version of AI is replaced with end-to-end, 360 AI? How little, in terms of human interests, objectives and motivations, is needed to sustain the production side of the economy? Perhaps the ambition and drive of a small team of tech bros can run the show. Hope not.
My experience with Claude Code is about 6 months old now, but I don't think we're close to having human mediated AI being replaced by 360 degree AI for most purposes.
When designing new systems, my experience was that AI almost continually goes off in weird directions, and you have to constantly reprompt it to get the results closer to what you want. It was frequently more effort than it was worth.
I *have* seen people just run tests continually against some code, with the test agents filing bug reports when crashes occur. And then another agent tries to fix those bugs. Even then, the fixes required close checking to see that they're reasonable.
But I don't think we're particularly close to these things running 24x7 without human monitoring. I'm not sure we're even seeing much progress in that specific direction.
That's very interesting. Maybe that ability to avoid spiraling off in unexpected directions is something that motivations, emotions and interests make us uniquely able to do. Of course we do sometimes move off topic but we're usually motivated to course correct. Also checking for 'reasonableness' or plausibility. Human judgement probably involves feelings. As AI reprocesses the entire context including the original prompt and its own responses every time it returns to the saddle, you'd have thought that this would keep it on track. But obviously not.