“We could simply be thinking about the benefits of intelligence wrong — arrogantly privileging the kind of mental tasks we humans happen to do especially well, while ignoring the value of the tasks we do poorly.”
This kind of “rhymes” with so much in the history of invention, where inventors trying and failing to “solve” one thing, accidentally invent (or inspire the invention) of a completely different thing.
I had a very similar thought - although I think in some ways saying 'arrogantly' is maybe even an "arrogant" error - perhaps worse, myopically blind to different modes as difficult to genuinely conceive.
"The universe is mostly the emergent behavior of complex systems" is a good definition of intelligence. The question is, has the complexity of computer systems risen to the level where any form of intelligence can have emerged. As near as I can tell computers and humans are still several orders of magnitude apart.
"it has very little to do with whether artificial intelligence is smarter than human intelligence in any sort of head-to-head matchup" - I maintain that this is an apples to oranges comparison. We are using the same word, "intelligence" to refer to completely different constructs.
"it’ll depend on its computer-like ability to hold huge amounts of data in its working memory and extract complex patterns from that data."
"Crucially, AI’s ability to do this doesn’t depend on its raw intelligence — only on its ability to handle huge amounts of data very quickly."
These last two statements nicely describe what AI tools really are, the ability to rapididly extract patterns from large amounts of data. Useful, yes. Make intelligent humans better at what they do, yes. Anything different than the abilty to mimic human thinking, no.
Definitions matter. Noah's writing is interesting because he manages to land on a description of reality while misusing words like intelligence, reason, understand, etc.
Very interesting reflection Noah, and for M. Chollet: "Increasing intelligence is not so much like "making the tower taller", it's more like "making the ball rounder". At some point it's already pretty damn spherical and any improvement is marginal." - the turn of phrase it's already pretty damn spherical is brilliant. and also quite analytically relevant to reflecting on the potential probability of diminishing returns on particularly areas where millions of years of biological evolution have likely in aggregate optimised us.
Your last observation: "We could simply be thinking about the benefits of intelligence wrong — arrogantly privileging the kind of mental tasks we humans happen to do especially well, while ignoring the value of the tasks we do poorly." - is particularly interesting although I think perhaps worse than 'arrogantly' - blindly and myopically might be better and more challenging. The effect of Not Knowing what We Dont' Know and not realising we don't know.
There is something quite intriguing on reflecting on cyborging (used only 1/3 seriously) our intelligence between what biologically optimisied from evolution versus what evolution either did not optiise for or didn't "care" about. But is at same time probably extremely hard for us to spot given our biological minds optimisation and blind spots (plus limited knowledge of such function).
ETA: Equally the observation on data, where in particular as I work in hard-asset financing in both of developed and developing markets, I find that people who come from working in Data Rich sectors in developed countries (as in e.g. the IT world) grossly and ongoing underestimate the degree to which data is limited for action in even developed world on at minimum hard-assets. More data collection of course is possibly - smart monitors but that's a lot of installations and while cost has collapsed, it ain't free (and needs to be human installed etc). But equally market operating logistics, especially in physical world and outside of best-resourced parts of developed world has material lmits. (it reminds me of the blind spots of app developers I have run into, like an app developer focused on "solving" African food logistics with an app (uber clone I suppose). And they default assumed the availability of the physical assets (storage, transport) and the quality of assets. Assumable for a US or a W. Euro market... not wise to assume elsewhere. :
"Why would intelligence top out like this? Well, if we think of intelligence as the ability to extract information from data, then even an infinitely advanced model endowed with infinite compute will be limited by the fact that there’s a limited amount of information that can be extracted from the data.
For one thing, data itself is in limited supply. You can’t transform the world unless you can (in some generalized sense) understand it, and you can’t understand the world unless you can measure it, and our ability to measure the world is inherently limited and finite"
I follow a lot on AI Noah (too much in fact) and while I’ve heard rumblings of this type of position, you’ve laid this out as clearly as I’ve ever read. Never thought you’d turn into one of my most influential figures on how I view AI, but this is excellent
The “third magic” post was great and that sort of knowledge outside of our perception (or where our limited perception prevents us from seeing/understanding) is an interesting concept.
To me AI’s two superpowers are knowledge (in the old days I knew a couple of bright kids who had practically memorized the encyclopedia- AI knows everything it trained on), which is not “intelligence”, and iteration (computing power).
Machine learning has been used in financial markets for decades and I’ve seen it uncover relationships humans wouldn’t have found (or looked for)….maybe unless you had a team of 1000 humans and have them broad remits.
Is knowledge plus iteration plus being able to filter iterative paths to the more promising ones (based upon past knowledge and finding overlaps) actually “intelligence”.?
So much to consider- just this morning this excellent article after an article about Ai detection on substack spurned a discussion that at times resembled the luddites tearing up weaving machines and so much guilt induced shame about how Ai users ripped up every available resource . Then Krugman on oligarchs in the USA . What it is to be human - not sure Claude will ever understand that.
“We could simply be thinking about the benefits of intelligence wrong — arrogantly privileging the kind of mental tasks we humans happen to do especially well, while ignoring the value of the tasks we do poorly.”
This kind of “rhymes” with so much in the history of invention, where inventors trying and failing to “solve” one thing, accidentally invent (or inspire the invention) of a completely different thing.
I had a very similar thought - although I think in some ways saying 'arrogantly' is maybe even an "arrogant" error - perhaps worse, myopically blind to different modes as difficult to genuinely conceive.
But I then want to cyborg myself (2/3 joke here)
"The universe is mostly the emergent behavior of complex systems" is a good definition of intelligence. The question is, has the complexity of computer systems risen to the level where any form of intelligence can have emerged. As near as I can tell computers and humans are still several orders of magnitude apart.
"it has very little to do with whether artificial intelligence is smarter than human intelligence in any sort of head-to-head matchup" - I maintain that this is an apples to oranges comparison. We are using the same word, "intelligence" to refer to completely different constructs.
"it’ll depend on its computer-like ability to hold huge amounts of data in its working memory and extract complex patterns from that data."
"Crucially, AI’s ability to do this doesn’t depend on its raw intelligence — only on its ability to handle huge amounts of data very quickly."
These last two statements nicely describe what AI tools really are, the ability to rapididly extract patterns from large amounts of data. Useful, yes. Make intelligent humans better at what they do, yes. Anything different than the abilty to mimic human thinking, no.
Definitions matter. Noah's writing is interesting because he manages to land on a description of reality while misusing words like intelligence, reason, understand, etc.
Very interesting reflection Noah, and for M. Chollet: "Increasing intelligence is not so much like "making the tower taller", it's more like "making the ball rounder". At some point it's already pretty damn spherical and any improvement is marginal." - the turn of phrase it's already pretty damn spherical is brilliant. and also quite analytically relevant to reflecting on the potential probability of diminishing returns on particularly areas where millions of years of biological evolution have likely in aggregate optimised us.
Your last observation: "We could simply be thinking about the benefits of intelligence wrong — arrogantly privileging the kind of mental tasks we humans happen to do especially well, while ignoring the value of the tasks we do poorly." - is particularly interesting although I think perhaps worse than 'arrogantly' - blindly and myopically might be better and more challenging. The effect of Not Knowing what We Dont' Know and not realising we don't know.
There is something quite intriguing on reflecting on cyborging (used only 1/3 seriously) our intelligence between what biologically optimisied from evolution versus what evolution either did not optiise for or didn't "care" about. But is at same time probably extremely hard for us to spot given our biological minds optimisation and blind spots (plus limited knowledge of such function).
ETA: Equally the observation on data, where in particular as I work in hard-asset financing in both of developed and developing markets, I find that people who come from working in Data Rich sectors in developed countries (as in e.g. the IT world) grossly and ongoing underestimate the degree to which data is limited for action in even developed world on at minimum hard-assets. More data collection of course is possibly - smart monitors but that's a lot of installations and while cost has collapsed, it ain't free (and needs to be human installed etc). But equally market operating logistics, especially in physical world and outside of best-resourced parts of developed world has material lmits. (it reminds me of the blind spots of app developers I have run into, like an app developer focused on "solving" African food logistics with an app (uber clone I suppose). And they default assumed the availability of the physical assets (storage, transport) and the quality of assets. Assumable for a US or a W. Euro market... not wise to assume elsewhere. :
"Why would intelligence top out like this? Well, if we think of intelligence as the ability to extract information from data, then even an infinitely advanced model endowed with infinite compute will be limited by the fact that there’s a limited amount of information that can be extracted from the data.
For one thing, data itself is in limited supply. You can’t transform the world unless you can (in some generalized sense) understand it, and you can’t understand the world unless you can measure it, and our ability to measure the world is inherently limited and finite"
Isn't AlphaFold already a good example of your thesis?
I follow a lot on AI Noah (too much in fact) and while I’ve heard rumblings of this type of position, you’ve laid this out as clearly as I’ve ever read. Never thought you’d turn into one of my most influential figures on how I view AI, but this is excellent
Bravo and thank you good sir
I feel like your 2nd and 3rd points are both variants of the "emergentism" argument
The “third magic” post was great and that sort of knowledge outside of our perception (or where our limited perception prevents us from seeing/understanding) is an interesting concept.
To me AI’s two superpowers are knowledge (in the old days I knew a couple of bright kids who had practically memorized the encyclopedia- AI knows everything it trained on), which is not “intelligence”, and iteration (computing power).
Machine learning has been used in financial markets for decades and I’ve seen it uncover relationships humans wouldn’t have found (or looked for)….maybe unless you had a team of 1000 humans and have them broad remits.
Is knowledge plus iteration plus being able to filter iterative paths to the more promising ones (based upon past knowledge and finding overlaps) actually “intelligence”.?
I’m not sure. Also not sure it matters.
Yes, cities do acquire new land, but they call it tower blocks.
Jazztazz.
So much to consider- just this morning this excellent article after an article about Ai detection on substack spurned a discussion that at times resembled the luddites tearing up weaving machines and so much guilt induced shame about how Ai users ripped up every available resource . Then Krugman on oligarchs in the USA . What it is to be human - not sure Claude will ever understand that.