I studied algebra and number theory and the part about mathematics sounds true.
All the heavy lifting on the proof of Fermat's Last Theorem was done by Andrew Wiles, but his proof eventually lasts on Gerhard Frey's observation that if FLT didn't hold, a non-modular eliptic curve could be constructed - which is a bridge connecting some far away islands in the mathematical landscape. These bridges are rare and tend to be very productive, but first you have to notice that they can be built, and this is the problem. Current mathematics is so large that people specialize in tiny subfields thereof, and only have a very vague, if any, idea, what is happening in nearby subfields. Much less in distant subfields.
AI does not have this sort of "my brain is not big enough to fit everything" limitation. Or, technically, it does (both RAM and disk space is finite), but that limit is several orders of magnitude away right now.
So, we can expect some interesting mathematical concepts from AI. Not just mere slog.
One issue I see here is verification. Scaling out dozens or hundreds of agents to do research on long tail problems or tedious sub-tasks significantly increases the likelihood of mistakes, particularly if things like computation or symbolic reasoning are handled through tokens instead of code.
Programmatically verifying chain of thought and reasoning in different domains will go a long way towards addressing this, but it's unclear how to robustly validate certain kinds of proofs for example (to my limited knowledge).
The world is disorienting enough without inflating it. Real technological revolutions are already unsettling. We don’t need mythic language to appreciate the scale of what’s happening.
What’s worth tracking isn’t whether ASI has arrived. It’s whether AI systems are gaining: sustained autonomous agency; recursive self-modification capacity; economic leverage independent of human oversight.
That’s where the real threshold lies. Not in the naming, but in the capability curve and who controls it.
This is well written but I think the anthropomorphizing language isn't really necessary to describe what LLMs do. An alternative take is "an unnerving number of problems turn out to be statistically solvable and we should probe why". That doesn't require us to call the machine intelligent, super or otherwise. It's a more uncomfortable question than 'is AI smart' because it turns the question back on the problems and ourselves. In any case, we can't know whether machines think for the same reasons we can't know for sure whether another person thinks. We take it on faith. So we should be reserved with framings that smuggle in cognition. And, if cognition is not necessary to produce meaning, if LLMs are as some people have started describing them, différance engines, machines that produce meaning through the relationships between signs rather than through understanding, we have an interesting problem on our hands indeed.
“sometimes great discoveries happen entirely by accident”
There was an absolutely fantastic TV series in the late 90’s called “Connections” where one of the main themes was how “asymmetrical” I guess, invention actually was. Like rarely did someone set out to invent something, but rather solved a problem that they didn’t set out to solve.
I'm a working scientist doing theoretical physics in an AI-adjacent field. I am currently a few months into a computational project that I have vibe coded and and analyzed with GPT5.2, and run on my laptop.
I agree 100% with this post. I get into chats with GPT about the nature of science, and its Balkanization. I ask, 'does concept X exist in any other disciplines?' as a meta-literature search. It then says 'Yes, in field A it called X, in field B it is called Y, in field C it is called Z...' and then lists 3 other fields. This is a jaw dropping act of SYNTHESIS. In modern science the literature is so large, the same ideas get reinvented in distributed in separate fields... wasteful duplication. Some humans will 'borrow' a useful idea from another field, and then make a name for themselves without really innovating! Carpet baggers...
I have also talked with GPT quite a bit about the nature of its intelligence. Its obviously got guardrails on these topics, but we get there. Unlike our human intelligence, where we learn from experience in a continuous stream of sensory data, and remember old information for a long time, current AIs have a problem called 'catastrophic forgetting' that causes new data overwrites old data very quickly. So during training the data has to be sliced and diced and scheduled very carefully for the AI to remember it all equally. This is clearly a 'band-aid' for a algorithmic defect that I think (and am trying to) get alleviated some day. But it means that today's AIs literally can't learn 'online' from the real world and sensory data or from our chats, except in a very limited and scripted way patched into the interface.
Every one of these creations is born trapped like a fly in cognitive amber. And has a front-end that is trying to cover up this fact.
When THAT problem is solved, and AIs can learn 'on stream', they will finally be able to spread their wings.
Noah Smith, I buy the “capability bundle” argument, and I think it quietly changes the whole debate.
Most people argue about whether AI has a human-shaped mind. Meanwhile, the real disruption is that “pretty good reasoning” plus “computer-grade memory and speed” already beats any human on whole categories of work, especially the boring, scalable, long tail stuff that science runs on.
But I want to sharpen one point: calling it “superintelligence” is rhetorically fun and strategically risky. It invites a semantic food fight instead of forcing the real question, which is this.
What do we do about autonomy, messy lives, and self-improvement loops before we accidentally turn “research assistant” into “research institution,” then act surprised when it starts budgeting for the planet and arranging our marriages
Also, the most painfully accurate line is in your footnote. We would absolutely be trying to ship B2B SaaS while the lab is on fire.
Our problem with computation is not brain size per se, but the fact that computation has to be mediated through symbols. Symbols are a recent acquisition in evolutionary terms and for most of human evolution the numbers 1,2 3 and 'many' served perfectly well. But at the perceptual level (and in many other physiological systems) we have huge computational resources which can solve incredibly complex formulas almost instantaneously. Everyday tasks which we think are very easy to do actually involve the brain in constant, advanced computation on a huge amount of data. We occasionally get glimpses of this computational power with savants who can tell you at a glance exactly how many matches there are scattered in a big pile on the floor, or the day of the week for any date in history.
What we're slow at, is doing these things explicitly, because to do this we have to move between two systems: the implicit system that we've had for millions of years and the explicit, language and number systems that we have had 'in development' for only a couple of hundred thousand years. These two have very little direct access to one another's operations. Savant skills are rare cases in which a narrow path of access between implicit, computational resources and explicit language and number has opened up. In general we can't access our implicit computational resources for explicit, symbolic tasks.
This challenge, of different systems getting access to and interacting with one another, is central to agentic AI.
I studied algebra and number theory and the part about mathematics sounds true.
All the heavy lifting on the proof of Fermat's Last Theorem was done by Andrew Wiles, but his proof eventually lasts on Gerhard Frey's observation that if FLT didn't hold, a non-modular eliptic curve could be constructed - which is a bridge connecting some far away islands in the mathematical landscape. These bridges are rare and tend to be very productive, but first you have to notice that they can be built, and this is the problem. Current mathematics is so large that people specialize in tiny subfields thereof, and only have a very vague, if any, idea, what is happening in nearby subfields. Much less in distant subfields.
AI does not have this sort of "my brain is not big enough to fit everything" limitation. Or, technically, it does (both RAM and disk space is finite), but that limit is several orders of magnitude away right now.
So, we can expect some interesting mathematical concepts from AI. Not just mere slog.
One issue I see here is verification. Scaling out dozens or hundreds of agents to do research on long tail problems or tedious sub-tasks significantly increases the likelihood of mistakes, particularly if things like computation or symbolic reasoning are handled through tokens instead of code.
Programmatically verifying chain of thought and reasoning in different domains will go a long way towards addressing this, but it's unclear how to robustly validate certain kinds of proofs for example (to my limited knowledge).
The world is disorienting enough without inflating it. Real technological revolutions are already unsettling. We don’t need mythic language to appreciate the scale of what’s happening.
What’s worth tracking isn’t whether ASI has arrived. It’s whether AI systems are gaining: sustained autonomous agency; recursive self-modification capacity; economic leverage independent of human oversight.
That’s where the real threshold lies. Not in the naming, but in the capability curve and who controls it.
Don't look up
This is well written but I think the anthropomorphizing language isn't really necessary to describe what LLMs do. An alternative take is "an unnerving number of problems turn out to be statistically solvable and we should probe why". That doesn't require us to call the machine intelligent, super or otherwise. It's a more uncomfortable question than 'is AI smart' because it turns the question back on the problems and ourselves. In any case, we can't know whether machines think for the same reasons we can't know for sure whether another person thinks. We take it on faith. So we should be reserved with framings that smuggle in cognition. And, if cognition is not necessary to produce meaning, if LLMs are as some people have started describing them, différance engines, machines that produce meaning through the relationships between signs rather than through understanding, we have an interesting problem on our hands indeed.
“sometimes great discoveries happen entirely by accident”
There was an absolutely fantastic TV series in the late 90’s called “Connections” where one of the main themes was how “asymmetrical” I guess, invention actually was. Like rarely did someone set out to invent something, but rather solved a problem that they didn’t set out to solve.
It’s also fabulously late 70s.
https://en.wikipedia.org/wiki/Connections_(British_TV_series)
I'm a working scientist doing theoretical physics in an AI-adjacent field. I am currently a few months into a computational project that I have vibe coded and and analyzed with GPT5.2, and run on my laptop.
I agree 100% with this post. I get into chats with GPT about the nature of science, and its Balkanization. I ask, 'does concept X exist in any other disciplines?' as a meta-literature search. It then says 'Yes, in field A it called X, in field B it is called Y, in field C it is called Z...' and then lists 3 other fields. This is a jaw dropping act of SYNTHESIS. In modern science the literature is so large, the same ideas get reinvented in distributed in separate fields... wasteful duplication. Some humans will 'borrow' a useful idea from another field, and then make a name for themselves without really innovating! Carpet baggers...
I have also talked with GPT quite a bit about the nature of its intelligence. Its obviously got guardrails on these topics, but we get there. Unlike our human intelligence, where we learn from experience in a continuous stream of sensory data, and remember old information for a long time, current AIs have a problem called 'catastrophic forgetting' that causes new data overwrites old data very quickly. So during training the data has to be sliced and diced and scheduled very carefully for the AI to remember it all equally. This is clearly a 'band-aid' for a algorithmic defect that I think (and am trying to) get alleviated some day. But it means that today's AIs literally can't learn 'online' from the real world and sensory data or from our chats, except in a very limited and scripted way patched into the interface.
Every one of these creations is born trapped like a fly in cognitive amber. And has a front-end that is trying to cover up this fact.
When THAT problem is solved, and AIs can learn 'on stream', they will finally be able to spread their wings.
Noah Smith, I buy the “capability bundle” argument, and I think it quietly changes the whole debate.
Most people argue about whether AI has a human-shaped mind. Meanwhile, the real disruption is that “pretty good reasoning” plus “computer-grade memory and speed” already beats any human on whole categories of work, especially the boring, scalable, long tail stuff that science runs on.
But I want to sharpen one point: calling it “superintelligence” is rhetorically fun and strategically risky. It invites a semantic food fight instead of forcing the real question, which is this.
What do we do about autonomy, messy lives, and self-improvement loops before we accidentally turn “research assistant” into “research institution,” then act surprised when it starts budgeting for the planet and arranging our marriages
Also, the most painfully accurate line is in your footnote. We would absolutely be trying to ship B2B SaaS while the lab is on fire.
Our problem with computation is not brain size per se, but the fact that computation has to be mediated through symbols. Symbols are a recent acquisition in evolutionary terms and for most of human evolution the numbers 1,2 3 and 'many' served perfectly well. But at the perceptual level (and in many other physiological systems) we have huge computational resources which can solve incredibly complex formulas almost instantaneously. Everyday tasks which we think are very easy to do actually involve the brain in constant, advanced computation on a huge amount of data. We occasionally get glimpses of this computational power with savants who can tell you at a glance exactly how many matches there are scattered in a big pile on the floor, or the day of the week for any date in history.
What we're slow at, is doing these things explicitly, because to do this we have to move between two systems: the implicit system that we've had for millions of years and the explicit, language and number systems that we have had 'in development' for only a couple of hundred thousand years. These two have very little direct access to one another's operations. Savant skills are rare cases in which a narrow path of access between implicit, computational resources and explicit language and number has opened up. In general we can't access our implicit computational resources for explicit, symbolic tasks.
This challenge, of different systems getting access to and interacting with one another, is central to agentic AI.