"Of course that leaves all sorts of other ideas to try — inventing newer hardware and ways of making hardware run faster, incorporating different kinds of data into LLMs, looking for newer and better algorithms, or even just investing more money."
The gains we are seeing with the open source LLMs, LlaMa, Vicuna, Wizard, et. al. with the 7B, 13B and 65B parameter models that can be run at home is where the action is. Spinning up your or your companies own custom LLM or large multimodal model (LMM) on run-of-the-mill GPUs or some AWS compute for a day or two is equivalent to the explosion of the WWW in 1996. As for running out of training data, Whisper by OpenAI, which offers an order of magnitude more training data by introducing voice, TV shows, radio broadcasts, YouTube videos, etc. into the mix will push back the "running out of training data based on text" by at least a year or two. Add in sparse neural nets and other technologies that introduce orders of magnitude more efficiency with current data sets, and the widespread adoption of custom hardware, and we have A.I. (M.L.) liftoff in all fields of human knowledge, including M.L. itself.
In his interview with ilya sutskever, Jensen Huang said he was looking forward to using hardware acceleration to make AI compute costs a million times cheaper.
Appears that there isna gentle 22 year linear fit until then of gentle manufacturing Capex increases. Maybe tied to long term GDP growth...
And it will be manufacturing that helps lower poverty. Because of their great living wages without a college degree.
Colleges got ridiculous and over their ROI skies partly because there was little alternative competition. When manufacturing middle class jobs go down and less a desirable life style, college was all the hype. Great lives are lived without a degree.
As a side note for your summary about limits to AI, another problem that was pointed out by AI researchers is that currently AI models could not understand logics, which is why without new methods to teach these models, as the amount of good training data dwindles the progress in generative AI will slow down. You could look into this article from Sydney Morning Herald, when ChatGPT was just published: https://www.smh.com.au/national/is-ai-coming-of-age-or-starting-to-reach-its-limits-20221213-p5c5uy.html.
Also, for your point of views about Millenials, I think that could be true in the US, and maybe continental Europe (with more young people's vote shifting right recently), but do you think it could be the same for young people in other English-speaking countries like Canada, UK or Australia? Here in Australia, I saw that many young people could not buy houses (or aspire to buy houses) anymore, so they are increasingly voting for left-wing parties: Labor, Green and independent candidates. In fact, that's partly the reason why the share in vote for 2 major parties in Australia is in a record low!
So if you come across any research papers or data for other Anglosphere countries, could you write more about whether Millennials there are still better off, and would their political ideas be more moderate?
You put into articulate words my exact thoughts on generative AI. It is unlikely we are going to see any modeling break-throughs, so at this point it is mostly a data arm race. Humans are the only source of new training data and are not supplying it any faster, so where are the bigger data sets going to come from now that the whole internet and the world’s libraries have been scoured?
These are already massive models where the marginal benefit of more parameters has probably been exhausted. I am doubtful that increases in computing power will help much. The computational power goes into the training and the training does not have to be very fast - slow release cycles seem to be working just fine. The generation of text on trained model requires comparatively little power.
Grad school lending is fundamentally different than UG lending. Lots of UGs pay for school through parents. Grad students don't meaning that schools are constrained in how much they can charge (or how small the stipend) by the student's total available credit.
For UGs schools can access parental credit (eg home equity loans) making it plausible that loans open up access to those whose parents lack that credit.
“A highly complex series of irrigation systems used water from the river for crops across the region, bolstering Ukraine’s reputation as one of the biggest food exporters in the world, with some 33 million hectares of farmland. But officials fear much of the region’s farmland may now be ruined.”
“The Ukrainian Agrarian Council estimates that the Kakhovka disaster could lead to a 14 percent drop in Ukraine’s grain exports. The country is the world’s fifth-largest exporter of wheat—meaning there will be serious knock-on effects for countries that rely on imports.”
And the surging wheat fields and wheat will be absorbing this toxic stew in the groundwater for years. Knock-on effects, indeed. The spark that set-off The Arab Spring was a serious bread shortage, which was caused by drought that collapsed the Ukrainian wheat harvest. An empty stomach is an angry stomach. Putin is now fighting with biological and chemical warfare via breaching this dam. He’d agreed to let Ukrainian wheat be sold and shipped, but that issue is now likely moot. I doubt China or Saudi Arabia are happy about this.
Throwing money at a market where supply is extremely inelastic tends to just raise prices. (We see the same problem with people wanting to subsidize rent in expensive markets like the Bay Area, without loosening regulations that prevent construction of new units.)
I work in AI research and I am strongly suspicious that there is still a ton of low-hanging fruit to be discovered in the "newer and better algorithms" space. While we may be close to hitting the limit for "how many parameters can we train at once", we are *nowhere near* hitting the limit for "how do we best arrange these parameters to get the best model?". I see new papers making 10-20% efficiency gains on this front on a monthly basis, and I see no reason to expect them to slow down. There has also been a renewed interest in totally new, moonshot ideas in AI, which makes me very excited. If the backpropagation gradient estimation algorithm was dethroned by a forward-only algorithm, for example, that could mean ~2x gains across the board for model training speed! So far, no such algorithm has been discovered, but many smart people are working on it, so I'm optimistic.
There should be nearly endless amounts of possible optimizations you can do for an LLM until it's not "large" anymore.
We don't know anything about how they work. What we do know is that transformer models are a very very generic architecture that probably learns a more specific architecture for the task. So, if you can specialize it again, it'll probably use less than all the power and memory ever, which is what it currently does.
Assume that AI improvements come to a screeching halt (very unlikely). We still have tremendous growth coming from taking advantage of it.
Think of electricity. They figured out how to generate it and make it ubiquitous. And we then spent decades of significant growth as we figured out how to best use it.
"Of course that leaves all sorts of other ideas to try — inventing newer hardware and ways of making hardware run faster, incorporating different kinds of data into LLMs, looking for newer and better algorithms, or even just investing more money."
The gains we are seeing with the open source LLMs, LlaMa, Vicuna, Wizard, et. al. with the 7B, 13B and 65B parameter models that can be run at home is where the action is. Spinning up your or your companies own custom LLM or large multimodal model (LMM) on run-of-the-mill GPUs or some AWS compute for a day or two is equivalent to the explosion of the WWW in 1996. As for running out of training data, Whisper by OpenAI, which offers an order of magnitude more training data by introducing voice, TV shows, radio broadcasts, YouTube videos, etc. into the mix will push back the "running out of training data based on text" by at least a year or two. Add in sparse neural nets and other technologies that introduce orders of magnitude more efficiency with current data sets, and the widespread adoption of custom hardware, and we have A.I. (M.L.) liftoff in all fields of human knowledge, including M.L. itself.
In his interview with ilya sutskever, Jensen Huang said he was looking forward to using hardware acceleration to make AI compute costs a million times cheaper.
Higher education seems to be costing more and delivering less, relative to free/cheap alternatives. How do we fix it?
Love the manufacturing huge upswing!
Appears that there isna gentle 22 year linear fit until then of gentle manufacturing Capex increases. Maybe tied to long term GDP growth...
And it will be manufacturing that helps lower poverty. Because of their great living wages without a college degree.
Colleges got ridiculous and over their ROI skies partly because there was little alternative competition. When manufacturing middle class jobs go down and less a desirable life style, college was all the hype. Great lives are lived without a degree.
Thank you for your newsletter Noah!
As a side note for your summary about limits to AI, another problem that was pointed out by AI researchers is that currently AI models could not understand logics, which is why without new methods to teach these models, as the amount of good training data dwindles the progress in generative AI will slow down. You could look into this article from Sydney Morning Herald, when ChatGPT was just published: https://www.smh.com.au/national/is-ai-coming-of-age-or-starting-to-reach-its-limits-20221213-p5c5uy.html.
Also, for your point of views about Millenials, I think that could be true in the US, and maybe continental Europe (with more young people's vote shifting right recently), but do you think it could be the same for young people in other English-speaking countries like Canada, UK or Australia? Here in Australia, I saw that many young people could not buy houses (or aspire to buy houses) anymore, so they are increasingly voting for left-wing parties: Labor, Green and independent candidates. In fact, that's partly the reason why the share in vote for 2 major parties in Australia is in a record low!
So if you come across any research papers or data for other Anglosphere countries, could you write more about whether Millennials there are still better off, and would their political ideas be more moderate?
You put into articulate words my exact thoughts on generative AI. It is unlikely we are going to see any modeling break-throughs, so at this point it is mostly a data arm race. Humans are the only source of new training data and are not supplying it any faster, so where are the bigger data sets going to come from now that the whole internet and the world’s libraries have been scoured?
These are already massive models where the marginal benefit of more parameters has probably been exhausted. I am doubtful that increases in computing power will help much. The computational power goes into the training and the training does not have to be very fast - slow release cycles seem to be working just fine. The generation of text on trained model requires comparatively little power.
In re solar panel production:
https://www.bbc.com/news/science-environment-65602519
Grad school lending is fundamentally different than UG lending. Lots of UGs pay for school through parents. Grad students don't meaning that schools are constrained in how much they can charge (or how small the stipend) by the student's total available credit.
For UGs schools can access parental credit (eg home equity loans) making it plausible that loans open up access to those whose parents lack that credit.
The Build-Some Things Country.
“A highly complex series of irrigation systems used water from the river for crops across the region, bolstering Ukraine’s reputation as one of the biggest food exporters in the world, with some 33 million hectares of farmland. But officials fear much of the region’s farmland may now be ruined.”
https://www.nbcnews.com/news/world/satellite-images-show-ukraine-dam-destruction-rcna88299?mc_cid=0f29d6a665&mc_eid=1a668c9ebc
https://www.wired.co.uk/article/kakhovka-dam-flooding-ukraine?mc_cid=0f29d6a665&mc_eid=1a668c9ebc
“The Ukrainian Agrarian Council estimates that the Kakhovka disaster could lead to a 14 percent drop in Ukraine’s grain exports. The country is the world’s fifth-largest exporter of wheat—meaning there will be serious knock-on effects for countries that rely on imports.”
And the surging wheat fields and wheat will be absorbing this toxic stew in the groundwater for years. Knock-on effects, indeed. The spark that set-off The Arab Spring was a serious bread shortage, which was caused by drought that collapsed the Ukrainian wheat harvest. An empty stomach is an angry stomach. Putin is now fighting with biological and chemical warfare via breaching this dam. He’d agreed to let Ukrainian wheat be sold and shipped, but that issue is now likely moot. I doubt China or Saudi Arabia are happy about this.
https://www.technologyreview.com/2023/06/01/1073799/the-world-is-finally-spending-more-on-solar-than-oil-production/amp/
Re: that student loan research -- have you read the Niskanen "Cost Disease Socialism" paper?
https://www.niskanencenter.org/cost-disease-socialism-how-subsidizing-costs-while-restricting-supply-drives-americas-fiscal-imbalance/
Throwing money at a market where supply is extremely inelastic tends to just raise prices. (We see the same problem with people wanting to subsidize rent in expensive markets like the Bay Area, without loosening regulations that prevent construction of new units.)
I work in AI research and I am strongly suspicious that there is still a ton of low-hanging fruit to be discovered in the "newer and better algorithms" space. While we may be close to hitting the limit for "how many parameters can we train at once", we are *nowhere near* hitting the limit for "how do we best arrange these parameters to get the best model?". I see new papers making 10-20% efficiency gains on this front on a monthly basis, and I see no reason to expect them to slow down. There has also been a renewed interest in totally new, moonshot ideas in AI, which makes me very excited. If the backpropagation gradient estimation algorithm was dethroned by a forward-only algorithm, for example, that could mean ~2x gains across the board for model training speed! So far, no such algorithm has been discovered, but many smart people are working on it, so I'm optimistic.
There should be nearly endless amounts of possible optimizations you can do for an LLM until it's not "large" anymore.
We don't know anything about how they work. What we do know is that transformer models are a very very generic architecture that probably learns a more specific architecture for the task. So, if you can specialize it again, it'll probably use less than all the power and memory ever, which is what it currently does.
Assume that AI improvements come to a screeching halt (very unlikely). We still have tremendous growth coming from taking advantage of it.
Think of electricity. They figured out how to generate it and make it ubiquitous. And we then spent decades of significant growth as we figured out how to best use it.