So it sounds as if we should start to worry less about technological stagnation and more about whether progress in AI is happening too quickly to keep it safe.
I was surprised to see that when Matt Yglesias writes about AI risk, he gets very strong pushback from a bloc of his paid subscribers who think the risk doesn't exist and that it's all Luddite nonsense.
It was hard to tell how many of those people were conservatives predisposed to this view by all the propaganda about global warming being a hoax, and how many were just going a little too far with the technophile, "abundance agenda" framework.
Either way it seems clearly wrong. People don't seem to appreciate that even if there's some theoretical argument for why there can't be existential AI risk, you can accept the argument with 99 percent confidence and still think AI risk reduction is a very, very high priority. Why is there so much resistance to this idea?
We've gotten a lot of the recent AI progress via a faster-than-exponential increase in the amount of computation used to train the models. That can't be sustained, and it stops being sustainable sooner if hardware isn't getting better at an exponential rate. There are ways around it -- specialized (neuromorphic/sparse/etc) hardware, algorithmic breakthroughs, quantum computing -- but if those don't pan out or take too long, it's reasonable to expect the current wave of progress to slow down.
Of course, there's enough innovation here already to disrupt quite a few industries even if progress stopped tomorrow.
I'm struck by what seems like a contradiction between this post and your more pessimistic recent post about decreasing future stock market returns. If this techno-optimisitc future with free unlimited energy, AI, biotech, robots, etc. is realized, couldn't all this free energy and other innovation free up a lot of $$ that could come back to us in stock returns? I take your point that the actual companies delivering innovation (solar panels, etc.) often realize little value themselves, but it seems our current civilization spends an awful lot of our resources on high cost energy extraction and use, healthcare, even housing etc. - all of which costs could potentially plunge through technological innovation.
‘For example, a team at the University of Sydney just claimed a breakthrough in sodium-sulfur batteries, which could be made without the relatively expensive metals (especially lithium) that go into lithium-iron batteries’
If I know anything about the Australian mining industry (& I do) given the recent discoveries of Lithium down here I’d expect BHP & Rio Tinto tanks & artillery to be surrounding U Syd to destroy these heretics quick smart 😀😀
Appreciated the mention of "massive desalination" through cheap and abundant energy. There are days when California has to pay states to take excess solar production, and I've always wondered why we're not putting that extra energy into desal. Perhaps the desal tech isn't there yet, but my understanding it that desals biggest hurdle is its energy intensity. Cheap, abundant solar can help us re-green areas becoming increasingly arid, like much of western US.
"If it extends the powers of our minds the way physical technology extended the powers of our bodies, another productivity boom is probably ahead." This is a great way of thinking about machine learning derived systems.
The steam engine cost a lot of people their jobs, but it benefited many others. It made male physical strength far less important to economic success -- enter the successful nerd. Allowing men to focus on their minds (and women to participate in labor) gave us a huge economic boom. However, it had other consequences. As a group, men are considerably physically weaker today than they were 200 years ago; some studies estimate that a American, colonial, frontier woman could probably defeat most males today in arm wrestling.
In the 19th century revolution, broader economic opportunities for weaker men lead to a productivity boom but also a physically weaker population, as economic (and reproductive) success became less tightly coupled from physical prowess. Women get 60% of all university degrees today, so today's machine learning revolution won't just affect males. If we were to mirror our experience of the first industrial revolution, we should expect this new one to result in an increase in a productivity boom as machine learning systems provide economic opportunities for the less intelligent. Considering how smart-biased our world has become, and how we treat and look down on those who are cognitively challenged, this ought to be a welcome thing.
The children's book author featured here is a perfect illustration. This guy is not smart / talented enough to produce and illustrate his own children's book. I don't mean to rag on him; I'm not either nor are most of us. However, AI allows this comparatively talentless hack to compete with professional author/ illustrators. This is the modern version of a John Henry duel. Expect a similar outcome: John Henry won his duel, but the steam engine eventually destroyed his industry. Should we expect AI to do the same to literature? Probably.
There is one more prediction that springs from this comparison though. Just as physical strength declined in the century of the steam engine, a corresponding decline in average IQ over the next century would not be unexpected. I doubt we'll get to Wall-E territory, but we might.
" if an autonomous robot uses AI to move around, is it predicting where it should go, or generating ideas of where to go?"
These are concretely two different approaches in robotics. Some approaches involve predicting which outcomes will succeed and taking them (i.e. solving a Bellman equation and then maximizing predicted utility) and others involve generating paths from a probability distribution, after conditioning on "evidence" that you were in fact successful in your goals.
Interestingly, though, there are derivations showing that "generate the most likely path, given I succeeded" and "solve the Bellman equation to find optimal path" end up being (mostly, up to important details) mathematically the same.
I think Noah should temper his glee over battery technology improving significantly in the near future. The Thermodynamics of batteries are well established. We know precisely the maximum amount of free energy available from any relevant electrochemical reaction. There are no magic bullets. Battery development is primarily in figuring out ways to create much higher, electrically connected, surface areas for reaction. The Lead Acid battery is completely dependent on the porous oxide growth that makes Pb:Acid batteries practical. Efforts to create similar structures in other battery chemistry reactions are dependent on expensive to make nano-structures. This will not change. There is no foreseeable battery technology that can compensate for the intermittent production of electricity from wind and solar at scale.
AI advancement will depend somewhat on the extension of Moore’s Law. Some think Moore’s Law is approaching a dead end. But ASML is working on a 2nm AUV lithography machine, as well as beginning work in pentameters and femtometers in its R&D labs.
"This means we’re not just going to save the planet from destruction"
I feel that you're *massively* downplaying climate change threats: yes the IEA revised it's estimates upwards, but we are still lagging behind on both annual investment needs and annual emission reductions.
From the same IEA report: "For electricity, in order to reach the installed capacity needed to generate 69% of electricity from renewables by 2030, average annual net additions need to be *30% higher for solar PV and more than twice as high for wind*.
👆 and that just for energy generation, but investments are also lagging behind on batteries, hydrogen, carbon capture, biofuels, etc. (see: (https://www.iea.org/reports/world-energy-investment-2022/overview-and-key-findings)). We are also miles away from the energy efficiency progress we should be making if we wanted to see the annual GDP/emissions decoupling of 7.5%.
So the IEA revising its estimates upwards does not mean we're going to save the planet from destruction (and anyways, the planet will be fine in long-run. We might just not be there to enjoy the views). The reality is that we're still lagging behind on every existing metric ...
Same thing with fusion - I agree it is exciting and that we should be optimists. But even the most optimists agree we won't have it in time for it to be meaningful at keeping temperature below 1.5/2 degrees.
When AlphaFold was released, I figured we’d have a long way to go before taking this technology for predicting protein folding — which is neat from a theoretical biology perspective — and adapting it for de novo protein design — which is a lot closer to the CAD-for-biology build-anything-on-earth-with-a-cellular-factory vision of a biotech driven future. But just this week, here’s a fantastic article on using a model like alphafold to design proteins! https://twitter.com/davejuergens/status/1601675072175239170?s=46&t=oF5U3APV379232QDZnUR1Q
Nuclear fission provides about 20% of the power in the United States. Nuclear provides about 75% of France's electricity. In an otherwise wide-ranging essay, ignoring fission is a glaring omission. The intermittency problem regarding solar and wind is far more significant than is acknowledged. Notably, with California's high solar penetration, grid integration is accomplished by inefficiently and intermittently dispatching natural gas-fired generation, calling into question the net environmental benefits of solar. Despite spending tens of billions on California solar since 2010, the U.S. EIA shows that since 2019, natural gas consumption for California electricity generation is climbing.
The proposed technical solution of batteries will not scale up to the required amounts for decades, if ever. At current prices, to build batteries to meet just 24 hours of California's electricity consumption will require more than twice the entire state of California budget. Those batteries will be in a perpetual replacement cycle since grid storage batteries last only 10 - 15 years at best. Finally, noting that the California transportation sector has the greatest amount of emissions of any sector, those batteries would best be reserved for transportation instead of displacing natural gas for electricity generation.
In summary, "Split, don't emit." To learn more, please visit the Californians for Green Nuclear Power website at CGNP dot org.
A nice synthesis of AI advances and biotech was published last week. Researchers use a diffusion model similar to DALL-E 2 to generate custom protein structures based on “prompts” in the form of geometric constraints. This allows for the creation of bespoke proteins to, for instance, coordinate with metals, or bind to ligands or other proteins. Protein sequence/structure state space is huge, so this is a difficult problem and consequently a huge finding. The generative models were validated with wet lab and structural experimental. This will be revolutionary in many ways, probably as important an advance as AlphaFold2 (indeed, it uses a similar model, called RoseTTAFold in its pipeline).
This is a "many-threaded" post that's provocative and useful.
One caution on techno-optimism, though. We should question what we describe as an "advance" and bear in mind it is not always without cost. We are always behind technological development by super-smart people at companies focused on innovation. We often need to play catch-up on monitoring and regulating some of technology's less desirable (read: frightening) long-tail effects.
Disruptions represented by technology can be industry-altering and remarkable; they can also be profoundly damaging--including to democratic institutions and our civil society. For example, who'd have thought Apple, which ran its famous 1984 Super Bowl commercial, would end up enabling Big Brother at the same time? (Feel like chilling your blood today? Read this: https://www.foreignaffairs.com/world/autocrat-in-your-iphone-mercenary-spyware-ronald-deibert).
I'm an Apple fan. I use Apple's products. (Who doesn't?!) But as Benedict Evans, whom you cite, has argued, when companies track you and seek to serve up content in line with your interests, Apple calls it an invasion of privacy. When Apple does the same thing, the company dubs it "personalization."
Ah, technology! And the interests to which it is put! Vaccine technologies for Covid-19 were life-saving, yes. But much of the technology used (mRNA) was already in place and had been for almost two decades. The speed of vaccine rollout was largely in removing regulatory barriers and accelerating approvals at the FDA.
Technology can improve lives and can save them. We should be grateful for such advances. But cutting-edge technology is a blade. If it's badly used, we bleed.
So it sounds as if we should start to worry less about technological stagnation and more about whether progress in AI is happening too quickly to keep it safe.
I was surprised to see that when Matt Yglesias writes about AI risk, he gets very strong pushback from a bloc of his paid subscribers who think the risk doesn't exist and that it's all Luddite nonsense.
It was hard to tell how many of those people were conservatives predisposed to this view by all the propaganda about global warming being a hoax, and how many were just going a little too far with the technophile, "abundance agenda" framework.
Either way it seems clearly wrong. People don't seem to appreciate that even if there's some theoretical argument for why there can't be existential AI risk, you can accept the argument with 99 percent confidence and still think AI risk reduction is a very, very high priority. Why is there so much resistance to this idea?
If you're looking for AI slowdown risks, the imminent end of Moore's law is a big one. (Don't believe me that that's happening? Nvidia's CEO thinks it is: https://www.marketwatch.com/story/moores-laws-dead-nvidia-ceo-jensen-says-in-justifying-gaming-card-price-hike-11663798618).
We've gotten a lot of the recent AI progress via a faster-than-exponential increase in the amount of computation used to train the models. That can't be sustained, and it stops being sustainable sooner if hardware isn't getting better at an exponential rate. There are ways around it -- specialized (neuromorphic/sparse/etc) hardware, algorithmic breakthroughs, quantum computing -- but if those don't pan out or take too long, it's reasonable to expect the current wave of progress to slow down.
Of course, there's enough innovation here already to disrupt quite a few industries even if progress stopped tomorrow.
I'm struck by what seems like a contradiction between this post and your more pessimistic recent post about decreasing future stock market returns. If this techno-optimisitc future with free unlimited energy, AI, biotech, robots, etc. is realized, couldn't all this free energy and other innovation free up a lot of $$ that could come back to us in stock returns? I take your point that the actual companies delivering innovation (solar panels, etc.) often realize little value themselves, but it seems our current civilization spends an awful lot of our resources on high cost energy extraction and use, healthcare, even housing etc. - all of which costs could potentially plunge through technological innovation.
‘For example, a team at the University of Sydney just claimed a breakthrough in sodium-sulfur batteries, which could be made without the relatively expensive metals (especially lithium) that go into lithium-iron batteries’
If I know anything about the Australian mining industry (& I do) given the recent discoveries of Lithium down here I’d expect BHP & Rio Tinto tanks & artillery to be surrounding U Syd to destroy these heretics quick smart 😀😀
Appreciated the mention of "massive desalination" through cheap and abundant energy. There are days when California has to pay states to take excess solar production, and I've always wondered why we're not putting that extra energy into desal. Perhaps the desal tech isn't there yet, but my understanding it that desals biggest hurdle is its energy intensity. Cheap, abundant solar can help us re-green areas becoming increasingly arid, like much of western US.
"If it extends the powers of our minds the way physical technology extended the powers of our bodies, another productivity boom is probably ahead." This is a great way of thinking about machine learning derived systems.
The steam engine cost a lot of people their jobs, but it benefited many others. It made male physical strength far less important to economic success -- enter the successful nerd. Allowing men to focus on their minds (and women to participate in labor) gave us a huge economic boom. However, it had other consequences. As a group, men are considerably physically weaker today than they were 200 years ago; some studies estimate that a American, colonial, frontier woman could probably defeat most males today in arm wrestling.
In the 19th century revolution, broader economic opportunities for weaker men lead to a productivity boom but also a physically weaker population, as economic (and reproductive) success became less tightly coupled from physical prowess. Women get 60% of all university degrees today, so today's machine learning revolution won't just affect males. If we were to mirror our experience of the first industrial revolution, we should expect this new one to result in an increase in a productivity boom as machine learning systems provide economic opportunities for the less intelligent. Considering how smart-biased our world has become, and how we treat and look down on those who are cognitively challenged, this ought to be a welcome thing.
The children's book author featured here is a perfect illustration. This guy is not smart / talented enough to produce and illustrate his own children's book. I don't mean to rag on him; I'm not either nor are most of us. However, AI allows this comparatively talentless hack to compete with professional author/ illustrators. This is the modern version of a John Henry duel. Expect a similar outcome: John Henry won his duel, but the steam engine eventually destroyed his industry. Should we expect AI to do the same to literature? Probably.
There is one more prediction that springs from this comparison though. Just as physical strength declined in the century of the steam engine, a corresponding decline in average IQ over the next century would not be unexpected. I doubt we'll get to Wall-E territory, but we might.
Posts like these are why I have a paid subscription. Thank you for another amazing post!
Can’t wait for the future.
" if an autonomous robot uses AI to move around, is it predicting where it should go, or generating ideas of where to go?"
These are concretely two different approaches in robotics. Some approaches involve predicting which outcomes will succeed and taking them (i.e. solving a Bellman equation and then maximizing predicted utility) and others involve generating paths from a probability distribution, after conditioning on "evidence" that you were in fact successful in your goals.
Interestingly, though, there are derivations showing that "generate the most likely path, given I succeeded" and "solve the Bellman equation to find optimal path" end up being (mostly, up to important details) mathematically the same.
I think Noah should temper his glee over battery technology improving significantly in the near future. The Thermodynamics of batteries are well established. We know precisely the maximum amount of free energy available from any relevant electrochemical reaction. There are no magic bullets. Battery development is primarily in figuring out ways to create much higher, electrically connected, surface areas for reaction. The Lead Acid battery is completely dependent on the porous oxide growth that makes Pb:Acid batteries practical. Efforts to create similar structures in other battery chemistry reactions are dependent on expensive to make nano-structures. This will not change. There is no foreseeable battery technology that can compensate for the intermittent production of electricity from wind and solar at scale.
AI advancement will depend somewhat on the extension of Moore’s Law. Some think Moore’s Law is approaching a dead end. But ASML is working on a 2nm AUV lithography machine, as well as beginning work in pentameters and femtometers in its R&D labs.
"This means we’re not just going to save the planet from destruction"
I feel that you're *massively* downplaying climate change threats: yes the IEA revised it's estimates upwards, but we are still lagging behind on both annual investment needs and annual emission reductions.
From the same IEA report: "For electricity, in order to reach the installed capacity needed to generate 69% of electricity from renewables by 2030, average annual net additions need to be *30% higher for solar PV and more than twice as high for wind*.
Clean energy investments should roughly triple from $1tn/annum today to $3tn every year for the next 30 years if we want to stay at 1.5 (i.e. we won't). And that's assuming we can (i.e. we have enough space, smart grids are developed, we have enough resources and space to build those RE power plants, etc.). (https://www.economist.com/leaders/2022/11/03/the-world-is-missing-its-lofty-climate-targets-time-for-some-realism)
👆 and that just for energy generation, but investments are also lagging behind on batteries, hydrogen, carbon capture, biofuels, etc. (see: (https://www.iea.org/reports/world-energy-investment-2022/overview-and-key-findings)). We are also miles away from the energy efficiency progress we should be making if we wanted to see the annual GDP/emissions decoupling of 7.5%.
Also, battery prices are going up because supply chains are not ready to match the growing demand, which itself is due to lack of supply chain investments. Supply chain underdevelopments risk derailing the energy transition (https://www.mckinsey.com/industries/oil-and-gas/our-insights/could-supply-chain-issues-derail-the-energy-transition).
So the IEA revising its estimates upwards does not mean we're going to save the planet from destruction (and anyways, the planet will be fine in long-run. We might just not be there to enjoy the views). The reality is that we're still lagging behind on every existing metric ...
Same thing with fusion - I agree it is exciting and that we should be optimists. But even the most optimists agree we won't have it in time for it to be meaningful at keeping temperature below 1.5/2 degrees.
Thanks for a great article.
When AlphaFold was released, I figured we’d have a long way to go before taking this technology for predicting protein folding — which is neat from a theoretical biology perspective — and adapting it for de novo protein design — which is a lot closer to the CAD-for-biology build-anything-on-earth-with-a-cellular-factory vision of a biotech driven future. But just this week, here’s a fantastic article on using a model like alphafold to design proteins! https://twitter.com/davejuergens/status/1601675072175239170?s=46&t=oF5U3APV379232QDZnUR1Q
It’s a remarkable time for technologists.
Nuclear fission provides about 20% of the power in the United States. Nuclear provides about 75% of France's electricity. In an otherwise wide-ranging essay, ignoring fission is a glaring omission. The intermittency problem regarding solar and wind is far more significant than is acknowledged. Notably, with California's high solar penetration, grid integration is accomplished by inefficiently and intermittently dispatching natural gas-fired generation, calling into question the net environmental benefits of solar. Despite spending tens of billions on California solar since 2010, the U.S. EIA shows that since 2019, natural gas consumption for California electricity generation is climbing.
The proposed technical solution of batteries will not scale up to the required amounts for decades, if ever. At current prices, to build batteries to meet just 24 hours of California's electricity consumption will require more than twice the entire state of California budget. Those batteries will be in a perpetual replacement cycle since grid storage batteries last only 10 - 15 years at best. Finally, noting that the California transportation sector has the greatest amount of emissions of any sector, those batteries would best be reserved for transportation instead of displacing natural gas for electricity generation.
In summary, "Split, don't emit." To learn more, please visit the Californians for Green Nuclear Power website at CGNP dot org.
A nice synthesis of AI advances and biotech was published last week. Researchers use a diffusion model similar to DALL-E 2 to generate custom protein structures based on “prompts” in the form of geometric constraints. This allows for the creation of bespoke proteins to, for instance, coordinate with metals, or bind to ligands or other proteins. Protein sequence/structure state space is huge, so this is a difficult problem and consequently a huge finding. The generative models were validated with wet lab and structural experimental. This will be revolutionary in many ways, probably as important an advance as AlphaFold2 (indeed, it uses a similar model, called RoseTTAFold in its pipeline).
This is a "many-threaded" post that's provocative and useful.
One caution on techno-optimism, though. We should question what we describe as an "advance" and bear in mind it is not always without cost. We are always behind technological development by super-smart people at companies focused on innovation. We often need to play catch-up on monitoring and regulating some of technology's less desirable (read: frightening) long-tail effects.
Disruptions represented by technology can be industry-altering and remarkable; they can also be profoundly damaging--including to democratic institutions and our civil society. For example, who'd have thought Apple, which ran its famous 1984 Super Bowl commercial, would end up enabling Big Brother at the same time? (Feel like chilling your blood today? Read this: https://www.foreignaffairs.com/world/autocrat-in-your-iphone-mercenary-spyware-ronald-deibert).
I'm an Apple fan. I use Apple's products. (Who doesn't?!) But as Benedict Evans, whom you cite, has argued, when companies track you and seek to serve up content in line with your interests, Apple calls it an invasion of privacy. When Apple does the same thing, the company dubs it "personalization."
Ah, technology! And the interests to which it is put! Vaccine technologies for Covid-19 were life-saving, yes. But much of the technology used (mRNA) was already in place and had been for almost two decades. The speed of vaccine rollout was largely in removing regulatory barriers and accelerating approvals at the FDA.
Technology can improve lives and can save them. We should be grateful for such advances. But cutting-edge technology is a blade. If it's badly used, we bleed.