It took me 7 years from the start of the dot com revolution until when I got my first email address. I was an early adopter because it took 8 more years for the world to get email addresses due to the mobile revolution. So for AI, this is only year 2! It’s still early to make judgements. Everyone is waiting for that “killer” app or device that changes the world. Of course it’s better to be that early adopter positioning yourself to be in front of the crowd when it comes!
The growth will come from increasing the $ per hour. Let's say the current ad model is $0.10 per hour, can you create a service that someone will pay $1 per hour?
A key difference between telecom/railways/etc. and the Internet is that on a telephone network only the telephone company can add "apps" and even those are fairly limited. You can basically transmit voice over a circuit switched network.
On the Internet, end users can add new behavior, and the limit is the limit of human imagination so the ceiling is high/unknown.
Argh Metcalfs law is BS, at best it’s something like n log n. You do not actually benefit a constant amount from being connected to every single person in the world.
Python now allows dictionary keys in double quotes inside curly braces in double quoted f-strings. That enough tech improvement to sustain me into 2025.
Regarding AI, we should distinguish between the foundation models from companies like OpenAI and Anthropic and all of the applications that will be built on top of them. Analogizing to the Internet, most of the money was made on all the applications sitting on top of the browser and server architecture, although these days browsers aren't a bad business either, assuming you can tie them to an ad business like Google does. AI applications can just make API calls or use an open source LLM, so they will have low fixed costs like other software businesses. Foundation models will be very capital intensive. Barriers to entry remain a question mark in both cases.
On AI, we'll probably see two very different types of companies:
- foundational model companies (openAI, anthropic, FB's llama)
- applied AI (Hebb.ia)
The first group are huge up front, possibly huge returns if nobody else can cheaply train a model like yours. This is you built the telegraph cables now you rake in the AT&T "telephone tax"
The second's just a classic software play on top of a new shiny tech called AI that unlocks productivity unachievable before, so the economics should be the same meaning applied AI companies will still have "geeks in a garage to millions" returns.
Maybe, BUT. Not all software is created equal. Desktop software and internet software both had very low variable costs -- so low that we could essentially think of software as free after the fixed costs of writing the software were complete. But AI inference costs a lot of money, at least right now. A lot of people expect that this cost will go to zero because...well, because they're used to thinking of software as something with no variable costs. But I don't think that'll happen. I think AI will continue to "light a pile of GPUs on fire" (as my friend Roon of OpenAI puts it) every time you use it. And that makes AI-powered software different from earlier generations.
1 - I doubt it will last long before becoming negligible again. The variable costs for a given performance metric go down by orders of magnitude every year for the same performance (https://semaphore.substack.com/p/the-cost-of-reasoning-in-raw-intelligence). Most applications won't need the latest & greatest AI, just something good enough to handle that app's messy data, so once it hits that point, their costs will become negligible in a couple of years.
2 - These costs are more like early cloud pricing which is monopoly pricing, not late cloud pricing which is commodity pricing, so prices should come down more.
3 - A lot of the variable cost is price of electricity and I think increase in supply due to solar + TBD tech will bring down prices faster than we can use it with AI.
All in all...AI's going to become a lot, lot cheaper to the point that most applied AI companies won't have a pricing difference. Of course you'll have your hedge funds and huge data companies using tons of AI, but they've always had huge cloud bills, so the economics should stay the same.
My conclusion is that we are going to need a higher saving rate to have enough investment to keep and improve on recent growth rates. This means more tax revenue from consumption taxes, especially progressive consumption taxes. Growth areas I expect will be computational biotech, lower cost zero-net CO2 emitting technologies and exports of capital equipment to developing countries
I've always had my doubts about the basic premise of AI large language models. The goal as I understand it is to learn everything, and my interactions with AI so far have convinced me that the achievement level in that direction is pretty mediocre.
We certainly don't think that productivity will be increased by every human trying to learn everything. Production is increased when we specialize. Clearly, it would be cheaper to produce an AI program that simply read mammograms well. And another one that "imagined" novel proteins and predicted their characteristics. And another that took the characteristics of existing mineral deposits and looked for similar conditions in other parts of the world.
The folks who conceived AI had a higher-than-average level of animal spirits and they seemed to have set their sights too broadly.
Knowing everything isn't actually the goal. AI as an industry doesn't really have goals. It's more based around trying things out and seeing what happens.
The reason LLMs appear to know "everything" is that nobody knows how to train a human level intelligence using current algorithms without feeding it monstrous amounts of data. If you don't do that you don't get a smart machine that knows less, the whole thing is just not at human level at all. So having so much knowledge is a side effect of the training process.
There's no law that says it has to be this way and in future it probably won't be. Noah says that AI "scales exponentially" which isn't quite true. Most of the work at the cutting edge in the past year has been around reducing costs and model sizes without sacrificing intelligence, not simply making bigger models. This is partly because the industry ran out of data (used it all) so nobody really knows what happens if you scale up again, nor is there any way to find out. But it's mostly because people realized how inefficient the current tech actually is. You don't really need to train LLMs on the entire internet if you make sure the input data is of much higher quality first.
Each of us, before specializing, learns a staggering amount of generalist knowledge. We learn to recognize objects, coordinate our muscles, speak at least one language, navigate the world, and build an intuition of how it works. This is the foundation in foundation models.
Modern LLMs operate on a mixture of experts framework, where different sections of the model specialize in different kinds of knowledge or tasks, and your query is routed to the 1-3 experts most confident they can answer the question. These experts can only be trained because they're built on the generalist foundation that learns the distribution of all human written (and increasingly recorded) works, and from this infers the hidden distribution of the real world that informed those works.
All this is to say the two are complimentary, not mutually exclusive!
One example of specialization is the reading of ancient manuscripts where letters have faded or portions of the writing have flaked off. If you can feed the letters of the language into an AI program, then AI can suggest possible restorations of the writing on the scroll. An expert human eye can determine which of the reconstructions makes the most sense. I don't see how LLM AI would be helpful in such a task.
I understand, but it's not so simple as 'feeding in the letters of the language'. If you have very few examples of this ancient writing and attempt to train a model on how to recognize it using only these few examples you'll see very poor performance. If, on the other hand, you take a model that's learned how to read most human languages (a generalist), and merely fine tune it on the small number of examples you have of ancient text, then assuming there is any statistical regularity across human language you'll achieve vastly better results. This is the power of foundation models, and why they have captivated the machine learning world for the last few years. You can achieve so much more by specializing an excellent generalist than by trying to train a specialist from random noise. The distance between an excellent generalist and a specialist is vastly smaller than the distance between a random network and a specialist, and the better the generalist the stronger this effect. The savings in cost and and time are measured in orders of magnitude.
Do not underestimate how much of your specialist task has already been learned by the generalist. The bulk of the task-useful information encoded in any human expert brain was learned before they ever began their specialist training. Placing an infant in medical school won't get you a better doctor!
In this case, a multimodal transformer is an excellent candidate for this task. Its character reconstruction estimates will be informed by a deep understanding of the statistical properties of language, essential to extracting meaning from noisy signals. It's the same reason you're so good at picking out what a speaker is saying in a noisy room, especially if you know them and the topic well. You predict as much as you sense, which allows you to disambiguate.
I promise we aren't all wasting our time with foundation models and transformers. Hype aside, they're really quite remarkably powerful tools!
I was simplifying drastically because I do not know what data they fed the program. However, I do know that such programs have produced impressive results. And each program must be focused on a specific ancient alphabetic script: Greek, Paleo Hebrew, block Hebrew, Syriac, etc.
MoE is actually not that popular; GPT4 supposedly used it, but every later model we know the architecture of is back to being one big blob.
Afaik the major issue is that we don't actually know what's in each "expert", because we basically don't know how models work internally. They weren't intentionally split into different topics or anything like that.
That's true. It's a nice story to suggest the experts in MoE map to real subject matter expertise, but I agree that whatever they split on is largely unintelligible to humans and certainly not anything as neat as different subject matters.
It's also true that I have no idea if Gemini or Claude uses MoE. We know about Mistral and suspect GPT-4, but it may prove unproductive in the future. I was merely trying to point out that the idea that specialization through MoE, LORAs, fine tuning, etc benefits from (and really in the modern context depends upon) generalization as a foundation.
I think you're right that LLMs can't live up to the hype, at least not for a while. But as a software developer working on AI for a law firm, I think there are a lot of problems related to reviewing large amounts of text that LLMs are reasonably good at helping with that traditional text processing methods aren't good at.
I'm inspired by the research paper INACIA: Integrating Large Language Models in Brazilian Audit Courts:
Opportunities and Challenges
by JAYR PEREIRA et al. Basically, the Brazilian government created a tool to make it easier to review complaints about how the government is spending money. The tool reviews the complaint, the relevant law, and the relevant contracts to create a recommendation for an auditor to review. This seems like a much better use case than chatbots.
Why I think the LLM's can never live up to the hype is that most of the language available to the model is super mediocre. Until the LLM is able to distinguish excellence, it can't be anything but mediocre. Thomas Friedman, whom I often respect, was gushing about an AI poem written about his wife's nonprofit. It was such puerile poetry that one could hardly believe it. It was based on the About section on the nonprofit’s website. Can you believe? The goal of poetry is to expose us in some way to something that we've never seen before. By definition, LLM cannot do that.
I have experienced that Bingo Copilot is a pretty good tool for searching the Internet. Bing copilot is set up to answer based on an Internet search rather than an LLM. Sometimes I have to rephrase my question to get the answer I'm looking for. If Bing copilot fails I try Google.
What I'm saying is that AI can often help with searches.
This isn't an either/or. All of the specific use cases will continue to get worked on as well, and they may or may not use LLMs. But there's also a limit to how much specialization makes sense. You wouldn't say you need "accounting brains" to do accounting - you just employ regular humans who learned accounting. Similarly, LLMs provide a baseline that can then be used for specific use cases, whether out of the box or with fine tuning for specialized problems.
LLMs weren't invented with a specific use case; OpenAI basically made a generic text generation model with all the text they could find, it sat there for a year-ish with not much hype, and then they made a chat website around it and everyone noticed it could answer anything they asked.
Since it's chat based it has a "universal interface"; you can't control what people ask so you have to be ready for anything. But there's also a principle called the "bitter lesson of machine learning" which says that a general model/architecture will always do everything better than a specific model/architecture, as long as you're willing to make it big enough. So the generalist people are continually obsoleting anyone who tries to make a specialist model.
Yeah my observation has been that if I ask it a question I already know the answer to it does pretty well but when I really need an answer because I’m working on something a bit tricky or obscure it tends to just make stuff up. So in a coding context it’s useful for spitting out boilerplate that I have trouble remembering, or showing me how to use some third party API I don’t have experience with, and in that sense it definitely helps with productivity, but I don’t see how it’s even close to replacing a skilled coder (though of course I would say that I guess).
My experience with chat AI is it sounds exactly like a college undergraduate who knows all the general opinions about everything but has no critical thinking about anything. I spent years as a college teacher, so I am familiar with this character.
“But there is one fundamental natural resource that these technologies use a lot of, and that’s time.”
The interesting thing about time is you can invest in it. It’s called wetware.
Ernest Hemingway was once asked in an interview if there was one critic he respected. His response: “Yes. Time.”
If one thing has proved of consistent value since the paintings on the walls of Chauvet Cave, it’s art. T.S. Eliot visited these paintings before beginning to write “The Wasteland.” You can see its influence in Picasso’s art.
I’ve done well in stock investments, but the investments in modern art and rare books for my heirs (aged 3 and 6 years) blow away any of my shareholder returns. If an investor chooses wisely, the value of a rare book or work of modern art is an investment in wetware. But the value of this wetware never declines and always goes up over time. I’m fact, it give greater returns over shorter periods of time with no down-side risk.
Two years ago, I purchased a First Edition copy of Hemingway’s in our time Three Mountains Press (1924). Only 170 copies were printed. A copy now sells at a minimum return of 54%. Rare wetware telescopes time. The book itself is a work of art. Time is not only in the title, but this is the Centennial Year for publication of this rare book, creating a spike in both interest and price. Funny how that works. The same holds true for works by Picasso and Dalí at any price tier. The beauty, so to speak, of an investment in wetware is that my heirs can inherit even greater returns because of time (12 years and 15 years) and not owe a single penny in taxes.
The point of this isn’t to brag. Like stocks, anybody can do this, it’s just easier, safer, and more rewarding with wetware. Like stocks, we all have the same information in re wetware. Anyone who reads can do this. Investing in hardware, stocks, etc. is much harder and riskier.
All too often we get lost in a high tech world and can be purblind to things of true lasting value.
By the way, a new mural by Banksy was discovered on a wall in southeast London.
AI seems cool but just does not seem very smart. It is completely unable to distinguish fact from fiction or sense from nonsense. It apparently suffers from a huge GIGO problem having been trained by scraping the internet, which includes huge amounts of of well, garbage, along with all of the useful knowledge it contains, but to AI is all just undifferentiated fodder. The AI responses now at the top of my google feed had failed until today to answer any question I had asked, and today's correct answer was a just a copy/paste of the first result (from Wikipedia) offered by the google algorithm, so no value added there.
That was true of the first one (GPT3.5), but most of the advances since then come from cleaning up the training data and adding high quality synthetic data.
The reason Google's AI answers are bad is that Google itself is bad these days, and they force the AI to quote the top search results.
How much did the 2022 hike in interest rates (driven by post-pandemic "revenge spending" along with the effects of the Ukraine war) help accelerate the enshittification of internet services?
Good question, but I don't think that was it. The economic issue for the tech industry is something called "R&D full expensing" that suddenly made employing engineers much more expensive recently. That and trend-following Elon firing all of Twitter is why they did layoffs.
Google also got worse because of improvements in content farm technology, replacing a bunch of the internals of the engine with AI they didn't understand (RankBrain, not generative AI), and because the search engine team was couped by the ads team a few years ago.
Major advances in technology over time have been a step function with extraordinarily large jumps followed by more gradual increases in or as humanity learns to adopt the new technology. So just because we can’t see the next jump in step function doesn’t mean it won’t happen.
Maybe it’s fusion, space travel, extending productive human life, quantum computing, frictionless materials, integrating the brain with neural networks. Or something we haven’t even imagined. But it will come and have amazing impact on human productivity.
At our stage of the tech cycle (adoption of the internet), most big investment returns like a Google or Amazon may have passed. But VCs make money by investing in better companies with better product market fit and evolve as the customers adopt and evolve. Still lots of 10x and 100x opportunities out there
Great note! My only quibble is on your Spotify/Amazon take, and your "all US businesses are online/have websites". AFAIK e-commerce penetration is only around 15% in the US - (https://fred.stlouisfed.org/series/ECOMPCTSA) much less than in comparable European countries. Its clearly more mature - but still has another 10 years probably to go.
It took me 7 years from the start of the dot com revolution until when I got my first email address. I was an early adopter because it took 8 more years for the world to get email addresses due to the mobile revolution. So for AI, this is only year 2! It’s still early to make judgements. Everyone is waiting for that “killer” app or device that changes the world. Of course it’s better to be that early adopter positioning yourself to be in front of the crowd when it comes!
There are only so many hours - very good point.
The growth will come from increasing the $ per hour. Let's say the current ad model is $0.10 per hour, can you create a service that someone will pay $1 per hour?
A key difference between telecom/railways/etc. and the Internet is that on a telephone network only the telephone company can add "apps" and even those are fairly limited. You can basically transmit voice over a circuit switched network.
On the Internet, end users can add new behavior, and the limit is the limit of human imagination so the ceiling is high/unknown.
That's why I think of software platforms like Instagram, Google, and Amazon as another infrastructure layer!
Argh Metcalfs law is BS, at best it’s something like n log n. You do not actually benefit a constant amount from being connected to every single person in the world.
OK that's fair. It's only locally true.
To put that in perspective, rainbows end when everyone owns a rainbow.
Python now allows dictionary keys in double quotes inside curly braces in double quoted f-strings. That enough tech improvement to sustain me into 2025.
Regarding AI, we should distinguish between the foundation models from companies like OpenAI and Anthropic and all of the applications that will be built on top of them. Analogizing to the Internet, most of the money was made on all the applications sitting on top of the browser and server architecture, although these days browsers aren't a bad business either, assuming you can tie them to an ad business like Google does. AI applications can just make API calls or use an open source LLM, so they will have low fixed costs like other software businesses. Foundation models will be very capital intensive. Barriers to entry remain a question mark in both cases.
On AI, we'll probably see two very different types of companies:
- foundational model companies (openAI, anthropic, FB's llama)
- applied AI (Hebb.ia)
The first group are huge up front, possibly huge returns if nobody else can cheaply train a model like yours. This is you built the telegraph cables now you rake in the AT&T "telephone tax"
The second's just a classic software play on top of a new shiny tech called AI that unlocks productivity unachievable before, so the economics should be the same meaning applied AI companies will still have "geeks in a garage to millions" returns.
Maybe, BUT. Not all software is created equal. Desktop software and internet software both had very low variable costs -- so low that we could essentially think of software as free after the fixed costs of writing the software were complete. But AI inference costs a lot of money, at least right now. A lot of people expect that this cost will go to zero because...well, because they're used to thinking of software as something with no variable costs. But I don't think that'll happen. I think AI will continue to "light a pile of GPUs on fire" (as my friend Roon of OpenAI puts it) every time you use it. And that makes AI-powered software different from earlier generations.
True, but:
1 - I doubt it will last long before becoming negligible again. The variable costs for a given performance metric go down by orders of magnitude every year for the same performance (https://semaphore.substack.com/p/the-cost-of-reasoning-in-raw-intelligence). Most applications won't need the latest & greatest AI, just something good enough to handle that app's messy data, so once it hits that point, their costs will become negligible in a couple of years.
2 - These costs are more like early cloud pricing which is monopoly pricing, not late cloud pricing which is commodity pricing, so prices should come down more.
3 - A lot of the variable cost is price of electricity and I think increase in supply due to solar + TBD tech will bring down prices faster than we can use it with AI.
All in all...AI's going to become a lot, lot cheaper to the point that most applied AI companies won't have a pricing difference. Of course you'll have your hedge funds and huge data companies using tons of AI, but they've always had huge cloud bills, so the economics should stay the same.
My conclusion is that we are going to need a higher saving rate to have enough investment to keep and improve on recent growth rates. This means more tax revenue from consumption taxes, especially progressive consumption taxes. Growth areas I expect will be computational biotech, lower cost zero-net CO2 emitting technologies and exports of capital equipment to developing countries
Noah, Constable Giggles wants you to get off the screen and pay him some attention.
Hehehe he went to sleep 🥰
And what did you put in his din din?
Pellets, greens, carrot bits...the usual!
He is loved.
As a founder of a largest tech job site in Ukraine this “focus on costs” angle is a good thing- at least in the medium term 😎
I've always had my doubts about the basic premise of AI large language models. The goal as I understand it is to learn everything, and my interactions with AI so far have convinced me that the achievement level in that direction is pretty mediocre.
We certainly don't think that productivity will be increased by every human trying to learn everything. Production is increased when we specialize. Clearly, it would be cheaper to produce an AI program that simply read mammograms well. And another one that "imagined" novel proteins and predicted their characteristics. And another that took the characteristics of existing mineral deposits and looked for similar conditions in other parts of the world.
The folks who conceived AI had a higher-than-average level of animal spirits and they seemed to have set their sights too broadly.
Knowing everything isn't actually the goal. AI as an industry doesn't really have goals. It's more based around trying things out and seeing what happens.
The reason LLMs appear to know "everything" is that nobody knows how to train a human level intelligence using current algorithms without feeding it monstrous amounts of data. If you don't do that you don't get a smart machine that knows less, the whole thing is just not at human level at all. So having so much knowledge is a side effect of the training process.
There's no law that says it has to be this way and in future it probably won't be. Noah says that AI "scales exponentially" which isn't quite true. Most of the work at the cutting edge in the past year has been around reducing costs and model sizes without sacrificing intelligence, not simply making bigger models. This is partly because the industry ran out of data (used it all) so nobody really knows what happens if you scale up again, nor is there any way to find out. But it's mostly because people realized how inefficient the current tech actually is. You don't really need to train LLMs on the entire internet if you make sure the input data is of much higher quality first.
well said.. a lot of the hype around "AI" is misplaced. Very much agree with the "scales exponentially" comment.
I agree with you and I especially appreciated the last sentence.
Each of us, before specializing, learns a staggering amount of generalist knowledge. We learn to recognize objects, coordinate our muscles, speak at least one language, navigate the world, and build an intuition of how it works. This is the foundation in foundation models.
Modern LLMs operate on a mixture of experts framework, where different sections of the model specialize in different kinds of knowledge or tasks, and your query is routed to the 1-3 experts most confident they can answer the question. These experts can only be trained because they're built on the generalist foundation that learns the distribution of all human written (and increasingly recorded) works, and from this infers the hidden distribution of the real world that informed those works.
All this is to say the two are complimentary, not mutually exclusive!
One example of specialization is the reading of ancient manuscripts where letters have faded or portions of the writing have flaked off. If you can feed the letters of the language into an AI program, then AI can suggest possible restorations of the writing on the scroll. An expert human eye can determine which of the reconstructions makes the most sense. I don't see how LLM AI would be helpful in such a task.
I understand, but it's not so simple as 'feeding in the letters of the language'. If you have very few examples of this ancient writing and attempt to train a model on how to recognize it using only these few examples you'll see very poor performance. If, on the other hand, you take a model that's learned how to read most human languages (a generalist), and merely fine tune it on the small number of examples you have of ancient text, then assuming there is any statistical regularity across human language you'll achieve vastly better results. This is the power of foundation models, and why they have captivated the machine learning world for the last few years. You can achieve so much more by specializing an excellent generalist than by trying to train a specialist from random noise. The distance between an excellent generalist and a specialist is vastly smaller than the distance between a random network and a specialist, and the better the generalist the stronger this effect. The savings in cost and and time are measured in orders of magnitude.
Do not underestimate how much of your specialist task has already been learned by the generalist. The bulk of the task-useful information encoded in any human expert brain was learned before they ever began their specialist training. Placing an infant in medical school won't get you a better doctor!
In this case, a multimodal transformer is an excellent candidate for this task. Its character reconstruction estimates will be informed by a deep understanding of the statistical properties of language, essential to extracting meaning from noisy signals. It's the same reason you're so good at picking out what a speaker is saying in a noisy room, especially if you know them and the topic well. You predict as much as you sense, which allows you to disambiguate.
I promise we aren't all wasting our time with foundation models and transformers. Hype aside, they're really quite remarkably powerful tools!
I was simplifying drastically because I do not know what data they fed the program. However, I do know that such programs have produced impressive results. And each program must be focused on a specific ancient alphabetic script: Greek, Paleo Hebrew, block Hebrew, Syriac, etc.
MoE is actually not that popular; GPT4 supposedly used it, but every later model we know the architecture of is back to being one big blob.
Afaik the major issue is that we don't actually know what's in each "expert", because we basically don't know how models work internally. They weren't intentionally split into different topics or anything like that.
That's true. It's a nice story to suggest the experts in MoE map to real subject matter expertise, but I agree that whatever they split on is largely unintelligible to humans and certainly not anything as neat as different subject matters.
It's also true that I have no idea if Gemini or Claude uses MoE. We know about Mistral and suspect GPT-4, but it may prove unproductive in the future. I was merely trying to point out that the idea that specialization through MoE, LORAs, fine tuning, etc benefits from (and really in the modern context depends upon) generalization as a foundation.
My concept is that we humans will be the generalists while specialists train AI on pattern recognition in very specific endeavors.
I think you're right that LLMs can't live up to the hype, at least not for a while. But as a software developer working on AI for a law firm, I think there are a lot of problems related to reviewing large amounts of text that LLMs are reasonably good at helping with that traditional text processing methods aren't good at.
I'm inspired by the research paper INACIA: Integrating Large Language Models in Brazilian Audit Courts: Opportunities and Challenges by JAYR PEREIRA et al. Basically, the Brazilian government created a tool to make it easier to review complaints about how the government is spending money. The tool reviews the complaint, the relevant law, and the relevant contracts to create a recommendation for an auditor to review. This seems like a much better use case than chatbots.
Why I think the LLM's can never live up to the hype is that most of the language available to the model is super mediocre. Until the LLM is able to distinguish excellence, it can't be anything but mediocre. Thomas Friedman, whom I often respect, was gushing about an AI poem written about his wife's nonprofit. It was such puerile poetry that one could hardly believe it. It was based on the About section on the nonprofit’s website. Can you believe? The goal of poetry is to expose us in some way to something that we've never seen before. By definition, LLM cannot do that.
The Brazilian app sounds like a great program.
I have experienced that Bingo Copilot is a pretty good tool for searching the Internet. Bing copilot is set up to answer based on an Internet search rather than an LLM. Sometimes I have to rephrase my question to get the answer I'm looking for. If Bing copilot fails I try Google.
What I'm saying is that AI can often help with searches.
This isn't an either/or. All of the specific use cases will continue to get worked on as well, and they may or may not use LLMs. But there's also a limit to how much specialization makes sense. You wouldn't say you need "accounting brains" to do accounting - you just employ regular humans who learned accounting. Similarly, LLMs provide a baseline that can then be used for specific use cases, whether out of the box or with fine tuning for specialized problems.
LLMs weren't invented with a specific use case; OpenAI basically made a generic text generation model with all the text they could find, it sat there for a year-ish with not much hype, and then they made a chat website around it and everyone noticed it could answer anything they asked.
Since it's chat based it has a "universal interface"; you can't control what people ask so you have to be ready for anything. But there's also a principle called the "bitter lesson of machine learning" which says that a general model/architecture will always do everything better than a specific model/architecture, as long as you're willing to make it big enough. So the generalist people are continually obsoleting anyone who tries to make a specialist model.
What I noticed about AI chat is that it couldn't answer every question I asked, which caused my great disillusionment with LLM's.
Yeah my observation has been that if I ask it a question I already know the answer to it does pretty well but when I really need an answer because I’m working on something a bit tricky or obscure it tends to just make stuff up. So in a coding context it’s useful for spitting out boilerplate that I have trouble remembering, or showing me how to use some third party API I don’t have experience with, and in that sense it definitely helps with productivity, but I don’t see how it’s even close to replacing a skilled coder (though of course I would say that I guess).
My experience with chat AI is it sounds exactly like a college undergraduate who knows all the general opinions about everything but has no critical thinking about anything. I spent years as a college teacher, so I am familiar with this character.
“But there is one fundamental natural resource that these technologies use a lot of, and that’s time.”
The interesting thing about time is you can invest in it. It’s called wetware.
Ernest Hemingway was once asked in an interview if there was one critic he respected. His response: “Yes. Time.”
If one thing has proved of consistent value since the paintings on the walls of Chauvet Cave, it’s art. T.S. Eliot visited these paintings before beginning to write “The Wasteland.” You can see its influence in Picasso’s art.
I’ve done well in stock investments, but the investments in modern art and rare books for my heirs (aged 3 and 6 years) blow away any of my shareholder returns. If an investor chooses wisely, the value of a rare book or work of modern art is an investment in wetware. But the value of this wetware never declines and always goes up over time. I’m fact, it give greater returns over shorter periods of time with no down-side risk.
Two years ago, I purchased a First Edition copy of Hemingway’s in our time Three Mountains Press (1924). Only 170 copies were printed. A copy now sells at a minimum return of 54%. Rare wetware telescopes time. The book itself is a work of art. Time is not only in the title, but this is the Centennial Year for publication of this rare book, creating a spike in both interest and price. Funny how that works. The same holds true for works by Picasso and Dalí at any price tier. The beauty, so to speak, of an investment in wetware is that my heirs can inherit even greater returns because of time (12 years and 15 years) and not owe a single penny in taxes.
The point of this isn’t to brag. Like stocks, anybody can do this, it’s just easier, safer, and more rewarding with wetware. Like stocks, we all have the same information in re wetware. Anyone who reads can do this. Investing in hardware, stocks, etc. is much harder and riskier.
All too often we get lost in a high tech world and can be purblind to things of true lasting value.
By the way, a new mural by Banksy was discovered on a wall in southeast London.
AI seems cool but just does not seem very smart. It is completely unable to distinguish fact from fiction or sense from nonsense. It apparently suffers from a huge GIGO problem having been trained by scraping the internet, which includes huge amounts of of well, garbage, along with all of the useful knowledge it contains, but to AI is all just undifferentiated fodder. The AI responses now at the top of my google feed had failed until today to answer any question I had asked, and today's correct answer was a just a copy/paste of the first result (from Wikipedia) offered by the google algorithm, so no value added there.
That was true of the first one (GPT3.5), but most of the advances since then come from cleaning up the training data and adding high quality synthetic data.
The reason Google's AI answers are bad is that Google itself is bad these days, and they force the AI to quote the top search results.
Oh is that why? I noticed that if I ask google “is Product X good?” it always answer “Yes it’s amazing!” by quoting from the Product X product page.
How much did the 2022 hike in interest rates (driven by post-pandemic "revenge spending" along with the effects of the Ukraine war) help accelerate the enshittification of internet services?
Good question, but I don't think that was it. The economic issue for the tech industry is something called "R&D full expensing" that suddenly made employing engineers much more expensive recently. That and trend-following Elon firing all of Twitter is why they did layoffs.
Google also got worse because of improvements in content farm technology, replacing a bunch of the internals of the engine with AI they didn't understand (RankBrain, not generative AI), and because the search engine team was couped by the ads team a few years ago.
Major advances in technology over time have been a step function with extraordinarily large jumps followed by more gradual increases in or as humanity learns to adopt the new technology. So just because we can’t see the next jump in step function doesn’t mean it won’t happen.
Maybe it’s fusion, space travel, extending productive human life, quantum computing, frictionless materials, integrating the brain with neural networks. Or something we haven’t even imagined. But it will come and have amazing impact on human productivity.
At our stage of the tech cycle (adoption of the internet), most big investment returns like a Google or Amazon may have passed. But VCs make money by investing in better companies with better product market fit and evolve as the customers adopt and evolve. Still lots of 10x and 100x opportunities out there
Great note! My only quibble is on your Spotify/Amazon take, and your "all US businesses are online/have websites". AFAIK e-commerce penetration is only around 15% in the US - (https://fred.stlouisfed.org/series/ECOMPCTSA) much less than in comparable European countries. Its clearly more mature - but still has another 10 years probably to go.