The essay was fascinating until the commentary by ChatGPT. I find that no matter what the subject, 1) My own conversations with GenAI are fascinating to me, 2) Anyone else's conversations with GenAI are incredibly boring. Is that the case for other people? If so, it suggests something about AI's limits.
I’ve seen a lot of comparison of this to dreams. It’s like we’ve built a machine that can dream for us, sometimes in ways that find solutions we need, sometimes in ways that confirm all our own biases, that make us think we’re on to something big when it’s just banal, but very often in ways that are utterly boring to anyone who doesn’t share our own thought background.
Regarding your First Magic, passing down/transmission/tradition for millennia hardly ever involved writing down. As a professional historian I must protest that technological processes are just about the last thing that ever gets written down. Instead, in a near universal pattern technological processes are passed down orally within a small group of people who safeguard the “secret" of their art for their chosen successors. Of course, these groups that try to control succession usually have some amount of leakage and the secrets of the craft spread, but almost always on oral rather than in written form.
It is interesting that many of these processes and inventions were written down for the first time to obtain a patent, which is another way of safeguarding technological secrets.
One day I will finish this slavery-and-abolition stack before Trump makes me but something like "The Reinvention Of Atlantic Slavery" is entirely post-Industrial Revolution and it was a kind of magic that you could write down how the machines worked and adapt them to tropical conditions they were not designed for. So the idea that you could write it down and in a very different time (the SCA) and place it could be invented again is a different enough concept from "science".
Of course it’s not just because people are trying to keep secrets - it’s because writing out everything that is needed to explain to someone who doesn’t already have your background knowledge and intuitions is *hard*. Look at an old cookbook that is trying to explain a standard recipe, and it says something like “add enough vinegar and rosemary, and then cook until it’s done”. The idea of measuring quantities and times wasn’t necessary when most people were just making variations on traditional recipes familiar in the culture, and when you could assume the person would try a few times to get the understanding that you can’t easily get from words, even with a modern recipe.
Also, in ancient and medieval times writing was the province of intellectuals--priests, bureaucrats. Until very very recently, most of the people who worked with their hands were not literate.
The stochastic nature of LLM responses is entirely by design. If you would prefer a determininiatic response, just set the temperature parameter to zero and it will regurgitate the same response each time.
I came to say precisely this. All the network parameters are deterministic, as well as the sequence of input tokens. That there is any randomness in the input-output relationship is a pure design choice, not an indispensable aspect of these systems.
Setting the temperature makes the results *repeatable* but not *predictable*, and I think it’s the latter quality that makes LLMs feel a bit like magic.
What's the difference between "repeatable" and "predictable"?
If temp=0, then I'd argue the output is *both* repeatable and predictable. The output is just the result of forward-propagating the input tokens through a fully determined network. Why would that not be predictable?
If your definition of predictable is that we could determine the output for a given set of input tokens *without* knowing the model weights. In that case, hardly any ML models would be "predictable", even very simple ones. Certainly, those inaccurate models don't feel like magic at all.
I think what you are really meaning to say is something along the lines of "modern LLMs are not *interpretable*". That's true, but again, it's also true of many, many older and far less capable models. It's not a secret weapon.
I've been training and deploying ML models in my business for around 13 years. Prior to that, I took grad-level ML courses while earning my PhD. It's entertaining for me that, nowadays, so many people without training or experience want to discuss these topics. Naturally, I frequently find myself correcting and explaining.
Yes, what I was calling predictable maps exactly onto interpretable. I’ll use the appropriate term in the future. And I’d never claim that that quality alone is a superpower — just that it’s part of what makes using an LLM feel like doing magic, where you get sometimes impressive results from a system that feels a lot more capricious than traditional computer software.
Your Facebook feed circa 2010 was already serving you ads based on non-interpretable ML models though. Did that feel like magic?
To me, what makes modern LLMs seem so magical is just that they are very capable. Earlier models were similarly uninterpretable, they were just less capable. The jump in a capability was driven largely by the scaling up of the training process to larger training data sets.
Small quibble: I think it's a mistake to say GPT is parametric but some ML models (especially the neural networks being referenced circa 2009 aka RNNs) aren't, because basically every machine learning model that can be stored and run by a computer is parametric (since it has to be in order to be stored as 1s and 0s).
IMHO neither you or Noah are correctly describing parametric v.s. non-parametric models.
A non-parametric model is not a model that has "no parameters", as you and Noah imply.
The opposite would then be that all models with parameters are parametric, which you then claim is necessarily all models stored on computers, but that's not correct.
A non-parametric model just doesn't have a *fixed number* of parameters. It still has parameters. Examples:
Non-parametric: decision trees, support vector machines. Neither of these has a fixed number of parameters. In decision trees, we can always grow additional subtrees, which increases the number of parameters. In support vector machines, we can add more support vectors to increase the number of parameters. Crucially, in both cases we are progressively refining both the number of parameters, and their values, during training/learning.
Parametric: Neural networks, linear models. Fixed number of parameters. The values of parameters evolves during training/learning, but the number of parameters does not change.
"Non-parametric models differ from parametric models in that the model structure is not specified a priori but is instead determined from data. The term non-parametric is not meant to imply that such models completely lack parameters but that the number and nature of the parameters are flexible and not fixed in advance."
Noah, you're an economist and it's hard to find a less public way to reach you, so I'll be direct and transactional. You have a large audience and are generally not stupid or crazy, and I want to offer you money to spend some time doing something you might not have done otherwise. Specifically, I want you to read a book and carefully consider its arguments (following up with the book's online resources if necessary). If you think the subject of the book is relevant to your Substack, I would also be willing to offer additional money to write a post about it, whether you agree with the book's thesis or not. (The authors might also be willing to join you on your podcast if you invited them, but that's not something I can influence.)
Assuming that the topic of the book is at least somewhat interesting, how much would I have to offer to pay to get you to consider accepting?
My question: would a future very advanced AI achieve the dream of Soviet central planners? Capitalism+free markets triumphed over Soviet-style socialism because government planners (even apart from the political pressure put on them) could never make decisions as effectively and efficiently as millions of decision-makers participating in the free market. But could AI? Could we reach a point where AI is so good that allowing it to plan all/most of economic activity would actually yield a more prosperous and more equitable economy? And if that were possible, then, well, should we let it?
You mention "emerging simplicity" as a possible outcome of complexity. This actually plays a fundamental role in our understanding of many physical phenomena. So it may be worth exploring in areas such as economics and sociology in parallel with the deployment of AI models.
A case in point is the concept of "quasiparticle" in many-body physics. For example, electrons in a metal cannot be understood as individual entities: there is many-body wave function that satisfies the Pauli principle and represents the interactions between zillions of electrons, blah blah blah. Yet at the end of the day you can describe the motion of electric current in a wire as a single "electron" pushed by an electric field, except that this "electron" doesn't have the known electron mass but a "renormalized" mass that can be computed from the fundamental exact theory. So the many-body complexity of the system is replaced by a system of non-interacting quasiparticles.
The quasiparticle approach has an obvious appeal for modeling human societies, and yet it doesn't seem that any serious attempt has been made to make the analogy more quantitative. I even see the concept as interesting in justifying taxation: we are all "renormalized" particles in a society, and therefore, what we make is not really "ours"...
Maybe there is an Uncertainty Principle in Neural networks. There can't simultaneously be explainabity and a solution to a complex systems question or something like that. This was a great essay. We struggle to understand how the brain works, it's not super that we struggle to understand how these systems work.
As we do not fully seem to understand how both AI and the human brain work (how do we get from everything we can see and measure in the brain to a thought in language?), my limited knowledge of AI is that its a lot of math and statistics. That makes me wonder: is getting a thought in my brain not also mainly math and statistics, though in an organic process vs digital? If that would be true, is there actually a fundamental difference between a human and an AI robot, and is the idea we are a someone just an illusion, or have I seen too much science fiction?
This is the idea that many scientists and philosophers have been wondering about in one form or another for centuries! Ever since Newton, some people like Laplace wondered if we might just be extremely complex clockwork mechanisms (and if you look at some of the 18th century automata you can see why it might be tempting: https://www.youtube.com/watch?v=ux2KW20nqHU )
In the 20th century as we got a better understanding of the difference between a clockwork mechanism and a general computer, we wondered if we might be computers.
Now we are having the thought that computational neural nets might be a better model - and they surely are. But there’s plenty of reason to believe that there might be some more breakthroughs that are needed before we get the full physical and mechanical understanding of a person.
(I would put it that way ”being a someone” isn't an illusion, but is just something that can be done in a mechanical way, without true magic.)
I continue to enjoy reposts in particular where they show your personal evolution of thought (the reality of people, not idée fixe". Also thought that the ChatGPT commentary was a great way to expand the discussion and wondered if you checked more than one AI to see if there was much overlap in their commentaries.
Thought provoking. Today the Nobel Prize in science (immunology) was announced for the elucidation of the gene allowing immune (T-cell) to recognize “self” and prevent autoimmune disease or, when the gene is disturbed, cause autoimmune disease; e.g. Type 1 diabetes. 30 years of research was honored in this award.
Would genAI facilitate these types of discoveries/innovations? Maybe by facilitating hypotheses which lead to the critical experiments?
Clearly, the chatGPT does an excellent job of editing & analyzing Noah’s creative thoughts.
Conceptualizing how gen AI will facilitate discoveries to improve living standards is hard for me to grasp. Help me understand.
Obligatory reference to Snow Crash and the "me" re "history" is probably not what ChatGPT was being asked but that it knew especially the AI literature well and how to soothe econometricians is a legitimately impressive thing
The essay was fascinating until the commentary by ChatGPT. I find that no matter what the subject, 1) My own conversations with GenAI are fascinating to me, 2) Anyone else's conversations with GenAI are incredibly boring. Is that the case for other people? If so, it suggests something about AI's limits.
I’ve seen a lot of comparison of this to dreams. It’s like we’ve built a machine that can dream for us, sometimes in ways that find solutions we need, sometimes in ways that confirm all our own biases, that make us think we’re on to something big when it’s just banal, but very often in ways that are utterly boring to anyone who doesn’t share our own thought background.
Regarding your First Magic, passing down/transmission/tradition for millennia hardly ever involved writing down. As a professional historian I must protest that technological processes are just about the last thing that ever gets written down. Instead, in a near universal pattern technological processes are passed down orally within a small group of people who safeguard the “secret" of their art for their chosen successors. Of course, these groups that try to control succession usually have some amount of leakage and the secrets of the craft spread, but almost always on oral rather than in written form.
It is interesting that many of these processes and inventions were written down for the first time to obtain a patent, which is another way of safeguarding technological secrets.
One day I will finish this slavery-and-abolition stack before Trump makes me but something like "The Reinvention Of Atlantic Slavery" is entirely post-Industrial Revolution and it was a kind of magic that you could write down how the machines worked and adapt them to tropical conditions they were not designed for. So the idea that you could write it down and in a very different time (the SCA) and place it could be invented again is a different enough concept from "science".
Of course it’s not just because people are trying to keep secrets - it’s because writing out everything that is needed to explain to someone who doesn’t already have your background knowledge and intuitions is *hard*. Look at an old cookbook that is trying to explain a standard recipe, and it says something like “add enough vinegar and rosemary, and then cook until it’s done”. The idea of measuring quantities and times wasn’t necessary when most people were just making variations on traditional recipes familiar in the culture, and when you could assume the person would try a few times to get the understanding that you can’t easily get from words, even with a modern recipe.
Also, in ancient and medieval times writing was the province of intellectuals--priests, bureaucrats. Until very very recently, most of the people who worked with their hands were not literate.
The stochastic nature of LLM responses is entirely by design. If you would prefer a determininiatic response, just set the temperature parameter to zero and it will regurgitate the same response each time.
I came to say precisely this. All the network parameters are deterministic, as well as the sequence of input tokens. That there is any randomness in the input-output relationship is a pure design choice, not an indispensable aspect of these systems.
Setting the temperature makes the results *repeatable* but not *predictable*, and I think it’s the latter quality that makes LLMs feel a bit like magic.
What's the difference between "repeatable" and "predictable"?
If temp=0, then I'd argue the output is *both* repeatable and predictable. The output is just the result of forward-propagating the input tokens through a fully determined network. Why would that not be predictable?
If your definition of predictable is that we could determine the output for a given set of input tokens *without* knowing the model weights. In that case, hardly any ML models would be "predictable", even very simple ones. Certainly, those inaccurate models don't feel like magic at all.
I think what you are really meaning to say is something along the lines of "modern LLMs are not *interpretable*". That's true, but again, it's also true of many, many older and far less capable models. It's not a secret weapon.
I've been training and deploying ML models in my business for around 13 years. Prior to that, I took grad-level ML courses while earning my PhD. It's entertaining for me that, nowadays, so many people without training or experience want to discuss these topics. Naturally, I frequently find myself correcting and explaining.
Yes, what I was calling predictable maps exactly onto interpretable. I’ll use the appropriate term in the future. And I’d never claim that that quality alone is a superpower — just that it’s part of what makes using an LLM feel like doing magic, where you get sometimes impressive results from a system that feels a lot more capricious than traditional computer software.
Fair enough.
Your Facebook feed circa 2010 was already serving you ads based on non-interpretable ML models though. Did that feel like magic?
To me, what makes modern LLMs seem so magical is just that they are very capable. Earlier models were similarly uninterpretable, they were just less capable. The jump in a capability was driven largely by the scaling up of the training process to larger training data sets.
ChatGPT writing is giving me ticks; just as enjoyable as going to a dental clinic- functional but not fun
Small quibble: I think it's a mistake to say GPT is parametric but some ML models (especially the neural networks being referenced circa 2009 aka RNNs) aren't, because basically every machine learning model that can be stored and run by a computer is parametric (since it has to be in order to be stored as 1s and 0s).
IMHO neither you or Noah are correctly describing parametric v.s. non-parametric models.
A non-parametric model is not a model that has "no parameters", as you and Noah imply.
The opposite would then be that all models with parameters are parametric, which you then claim is necessarily all models stored on computers, but that's not correct.
A non-parametric model just doesn't have a *fixed number* of parameters. It still has parameters. Examples:
Non-parametric: decision trees, support vector machines. Neither of these has a fixed number of parameters. In decision trees, we can always grow additional subtrees, which increases the number of parameters. In support vector machines, we can add more support vectors to increase the number of parameters. Crucially, in both cases we are progressively refining both the number of parameters, and their values, during training/learning.
Parametric: Neural networks, linear models. Fixed number of parameters. The values of parameters evolves during training/learning, but the number of parameters does not change.
From Wikipedia: (https://en.wikipedia.org/wiki/Nonparametric_statistics#Non-parametric_models)
"Non-parametric models differ from parametric models in that the model structure is not specified a priori but is instead determined from data. The term non-parametric is not meant to imply that such models completely lack parameters but that the number and nature of the parameters are flexible and not fixed in advance."
Until unaligned AI kills everyone.
Noah, you're an economist and it's hard to find a less public way to reach you, so I'll be direct and transactional. You have a large audience and are generally not stupid or crazy, and I want to offer you money to spend some time doing something you might not have done otherwise. Specifically, I want you to read a book and carefully consider its arguments (following up with the book's online resources if necessary). If you think the subject of the book is relevant to your Substack, I would also be willing to offer additional money to write a post about it, whether you agree with the book's thesis or not. (The authors might also be willing to join you on your podcast if you invited them, but that's not something I can influence.)
Assuming that the topic of the book is at least somewhat interesting, how much would I have to offer to pay to get you to consider accepting?
(If it matters, the actual book itself is this one: https://tinyurl.com/2s4epwp6 )
Great post; absolutely fascinating.
My question: would a future very advanced AI achieve the dream of Soviet central planners? Capitalism+free markets triumphed over Soviet-style socialism because government planners (even apart from the political pressure put on them) could never make decisions as effectively and efficiently as millions of decision-makers participating in the free market. But could AI? Could we reach a point where AI is so good that allowing it to plan all/most of economic activity would actually yield a more prosperous and more equitable economy? And if that were possible, then, well, should we let it?
You mention "emerging simplicity" as a possible outcome of complexity. This actually plays a fundamental role in our understanding of many physical phenomena. So it may be worth exploring in areas such as economics and sociology in parallel with the deployment of AI models.
A case in point is the concept of "quasiparticle" in many-body physics. For example, electrons in a metal cannot be understood as individual entities: there is many-body wave function that satisfies the Pauli principle and represents the interactions between zillions of electrons, blah blah blah. Yet at the end of the day you can describe the motion of electric current in a wire as a single "electron" pushed by an electric field, except that this "electron" doesn't have the known electron mass but a "renormalized" mass that can be computed from the fundamental exact theory. So the many-body complexity of the system is replaced by a system of non-interacting quasiparticles.
The quasiparticle approach has an obvious appeal for modeling human societies, and yet it doesn't seem that any serious attempt has been made to make the analogy more quantitative. I even see the concept as interesting in justifying taxation: we are all "renormalized" particles in a society, and therefore, what we make is not really "ours"...
Thank you Noah
Maybe there is an Uncertainty Principle in Neural networks. There can't simultaneously be explainabity and a solution to a complex systems question or something like that. This was a great essay. We struggle to understand how the brain works, it's not super that we struggle to understand how these systems work.
As we do not fully seem to understand how both AI and the human brain work (how do we get from everything we can see and measure in the brain to a thought in language?), my limited knowledge of AI is that its a lot of math and statistics. That makes me wonder: is getting a thought in my brain not also mainly math and statistics, though in an organic process vs digital? If that would be true, is there actually a fundamental difference between a human and an AI robot, and is the idea we are a someone just an illusion, or have I seen too much science fiction?
This is the idea that many scientists and philosophers have been wondering about in one form or another for centuries! Ever since Newton, some people like Laplace wondered if we might just be extremely complex clockwork mechanisms (and if you look at some of the 18th century automata you can see why it might be tempting: https://www.youtube.com/watch?v=ux2KW20nqHU )
In the 20th century as we got a better understanding of the difference between a clockwork mechanism and a general computer, we wondered if we might be computers.
Now we are having the thought that computational neural nets might be a better model - and they surely are. But there’s plenty of reason to believe that there might be some more breakthroughs that are needed before we get the full physical and mechanical understanding of a person.
(I would put it that way ”being a someone” isn't an illusion, but is just something that can be done in a mechanical way, without true magic.)
I continue to enjoy reposts in particular where they show your personal evolution of thought (the reality of people, not idée fixe". Also thought that the ChatGPT commentary was a great way to expand the discussion and wondered if you checked more than one AI to see if there was much overlap in their commentaries.
Thought provoking. Today the Nobel Prize in science (immunology) was announced for the elucidation of the gene allowing immune (T-cell) to recognize “self” and prevent autoimmune disease or, when the gene is disturbed, cause autoimmune disease; e.g. Type 1 diabetes. 30 years of research was honored in this award.
Would genAI facilitate these types of discoveries/innovations? Maybe by facilitating hypotheses which lead to the critical experiments?
Clearly, the chatGPT does an excellent job of editing & analyzing Noah’s creative thoughts.
Conceptualizing how gen AI will facilitate discoveries to improve living standards is hard for me to grasp. Help me understand.
Obligatory reference to Snow Crash and the "me" re "history" is probably not what ChatGPT was being asked but that it knew especially the AI literature well and how to soothe econometricians is a legitimately impressive thing
It’s like protoculture from Robotech.
Nobody understands how it works, but it works. And it gives us a future full of awesome robots and warp drives and big hair.
I’m so here for it.