I'm an older working R1 research scientist in an AI-adjacent field.
This post rings true to me. And it boils down to one thing IMO, learned through a long career in science. Meaningful intellectual progress, making new insights and understanding, is exponentially hard. That means progress is logarithmic in time-integrated effort.
Popular history gets this wrong... it's all Eureka! a lone genius has a brainwave and the world is changed forever. That is the numerator, the insight. The denominator is how many ideas were abandoned, how many researchers didn't have the insight, how many students helped push, but then flew away to industry, how many research grants ended up making only mediocre papers?
And the language around ASI assumes, all we have to do is build a robot Einstein, pump in some electrons and its Eureka's every day, or a rapidly rising hyperbolic crescendo of Eureka's to a finite time singularity.
To me, that expectation is risable. We have forgotten Gödel.
Every past breakthrough, no matter how large, has just pushed back the frontier of human knowledge a little more.
The frontier labs are not building AI Einstein's that can make breakthroughs. We are building tools for an AI-powered scientific apparatus, with humans embedded in it for at least a while longer. And that scientific apparatus will make the same plodding, occasional small jump forward progress as before. And when AI really helps move the enterprise forward, it will just be a subtle but real trend break on the scientific progress curve from 2020 to 2040.
From the introduction: "Ramez’s case necessarily rests on a lot of assumptions"
I think the implication here is backwards. It is Ramez, in contrast to the AI doomers, that tries to avoid assumptions (especially those with massive error bars). Every AI doomer "50/50 we die" post I read makes implausible assumptions with simple slight of hand.
The one phrase everyone needs to take from this lengthy post is "Diminishing Returns". In retrospect it will seem obvious, that the AI focused catch up in hardware, firmware, infrastructure, compilers, software, and platforms compounded to create the impression that there were no diminishing returns.
Ramez shows us it is an illusion. (And most using these tools for production work experience it whether they realize it or not).
This all reminds me of the digital camera "megapixel race" of the first decade of the millennium. We started the millennium with 1-2MP cameras and exited it with closer to 20MP cameras -- a 10x increase!
The gains in visual fidelity were incredible! however photographers quickly realized their SD cards from just 2 years ago were unable to hold more than a few pictures, their computers could not handle the processing requirements of these larger photos, and at these higher resolutions, only the best lenses could actually take advantage of the resolution.
Much like AI the raw performance numbers hid the true story. In orderto double the size of a picture, a 2D matrix, you and to quadruple its pixel count. Much like AI, the leaps in technology just pushed the productivity bottleneck somewhere else, somewhere with mature already-highly-optimized technology.
Today, almost 2 decades later, the best professional digital still cameras have "only" about 60 megapixels, most video cameras are capped at 12 megapixels (4K).
Stories like these echo throughout the the last 2 centuries. Why is it that we think this time, bloggers with little to no computer or software expertise, have good reason to believe, this time is different?
Faith in the collective precognitive future given us by a century of Science Fiction writers. That have accurately mapped out specifics of future technological innovation in quite magical and disturbing ways. Flash Gordon. Dick Tracy, Jules Verne--the list goes on.
Regarding genuine machine consciousness, I personally, was sold at age twelve after reading Heinlein's "The Moon is a harsh mistress."
This is a great post! I wanted to build on the research being harder than the benchmarks and forecasts.
People with management experience know how hard it can be to specify and direct what needs to get done, and how this is qualitatively different than just executing on a task. Specification itself is hard and doesn't always have just one solution.
Benchmarks necessarily do some of the specification work. For people who haven't seen it yet, go look at the OpenAI GDPval prompts. This is meant to model performance on economically valuable, real-world tasks. The prompt is very clearly doing the specification work that wouldn't be done so neatly "in the wild".
Verifiable domains like math are then where AI excels because the verifier *is* the spec.
Ashby's Law of Requisite Variety says that only variety can absorb variety: for a system to successfully control or regulate another (or its environment), it needs to be able to tackle a greater variety of states/responses than what its controlling.
You can think of specification as this regulation done in advance: it anticipates the ways a task can go wrong and constrains the work against them.
Human accumulate variance through experience, and the collaborative process within organizations are also part of the variety that can absorb variety. This is why METR says that full automation will require more "foresight, prediction, creating one’s own feedback loops" but I think it also depends not just on AI feedback loops, but AI-human feedback loops. That's part of the variety process, and why I'm skeptical of reaching Type 4 (humans add no research value)
Another way to get that variety is to release world model fragments on the real world.
Then, the central (MoE) model can call up a number of specialised human and AI experts, who will necessarily contradict each other on complicated topics. This will enable the central model to test, stress test, experiment, not necessarily with a single outcome as complicated dynamic systems in the real world are often non-linear. But it could connect a range of possible outcomes to necessary conditions, paths and criteria for discernment.
The downside of an intelligence tempered by and resilient to real life conditions would be the utter uselessness of benchmarks to test it.
Yes! and if its released into the world, its then at the pace of real world feedback. Maybe then the judgement moves from benchmarks to how we evaluate people - through track record
A lot of what’s going on in this post involves attributing constant returns to ECI, or some simple function of returns to ECI. Why should we think that ECI is measured on a meaningful scale that can support this sort of inference, rather than being a mere ordinal scale like the Mohs scale of hardness, or something with logarithmic or exponential features?
Virtually all interesting problems exhibit exponentially-growing search spaces. That is, if you search ahead to depth d, the number of candidate solutions that must be explored grows as exp(d). A search heuristic must prune an exponentially-large subspace in order to deepen the search space from d to d+1. RSI requires finding a never-ending sequence of exponentially-effective search heuristics.
Let's consider the generate-and-test style of RL with verified rewards. The LLM is the generator of candidate solutions and the verifier tests them for correctness. One form of intelligence is the ability to extract a constraint from the verifier and incorporate it into the generator so that it stops generating solutions that would fail that constraint (or, conversely, only generates solutions that satisfy that constraint).
What limits this form of intelligence? First, the existence of constraints that can be extracted from the test. Second, the ability to incorporate those constraints into the generator. The theory of computational complexity studies these limits. If P != NP, then there are many problems (the NP-Complete problems) where there is no way to extract constraints that are exponentially effective. If P != NP, RSI cannot occur for such problems.
I believe that this methodology would have given you a significantly lower value than 5-10x if you had run the same methodology a year earlier, and we should expect it to increase over time as AI models get stronger and automate more of the research loop. Do you agree with this?
One argument for this is that if you kept subtracting ECI points from the models, the humans' productivity wouldn't exponentially fall towards 0. It would stabilize at a fairly reasonable number without AI assistance, and at that point, the productivity gain per ECI point would be ~0%. Your estimates suggest that the AIs have now started feeding back into the RSI loop enough to make this number increase to 3% per ECI point and I would expect it to keep increasing as they automate more and more of the research process.
To assess how close we are to a self-sustaining RSI loop, we'd ideally like to be able to measure how this number is trending over time. It's plausible to me that this trend would look pretty fast though I'm not sure either way.
(x-posted this comment from the other version of the post.)
One small comment with regard to Opus 5.5 - I only use it for agentic coding, but it is visibly much more economical with tokens. Maybe 3x as economical as 5.0 was. You get a lot more bang for the same token budget. In this particular sense, it wasn't a modest marginal improvement.
I don't think super intelligence will come soon, either. Reading this, it makes me think that intelligence has an innate cost. The human brain weighs 1300 grams, a humming bird's weighs 0.13 games. Humans are clearly far more intelligent than hummingbirds, but 10000x? The math in the article also shows there is exponential resources required to sustain linear capability growth.
Part of intelligence is being able to draw conclusions across different pieces of seemingly unrelated information. However, this is a network effect, which is exponential. There is no magic sauce which can join together infinitely large information spaces. As we learn more about intelligence, I believe we will discover a power law, similar to the inverse square law for energy dissipation. Every level of intelligence requires a certain amount of cross connectivity, and capability cannot grow beyond that. Exponential takeoff is impossible.
Interestingly, bird brains have a more efficient architecture than our brains, which partly accounts for how smart birds are relative to their brain size. AI engineering has to hope that increasing efficiency for reduction of compute will continue. This doesn't take away from your good point but I think both AI engineers and biologists have been formulating laws about this kind of scaling already. I also wonder if recording history, writing symphonies and contemplating your finitude is not in some ways 10000x smarter than the hummingbird...
Like Uwe, I would say that birds have really good brains. Corvids like crows and magpies have about 10 g of brain matter and yet they are very, very, very smart - assessed to be about as intelligent as 3-5 year old human kids. That would indicate that their brains may be 30x as efficient as ours.
when folks first built automatic looms that replaced skilled human labor, they imagined humans as automata that could work on a similar principle.
when folks first build computers that could automate many computational tasks, they imagined the brain was a big computer, and imagined building a computer brain that would work on a similar principle.
when folks first build overparameterized deep learning systems that could really absorb and make expressive or generative extrapolations of huge uncurated datasets, they imagined the brain as a deep learning system that worked on a similar principle.
obviously each stage of this progression is 'more correct' than the one before it. But how do we know that the last proposition is 'effectively correct' or even effectively complete?
I think we are clearly getting closer (closing the frontier), but might still be missing a little secret sauce. One bit of evidence i have for that is the difference between ANNs that have exponential memory (aka catastrophic forgetting) and which requires careful batch training and massive engineering around training schedules, versus biological NNs, which (somehow) have power-law forgetting (with a rather small exponent)!!
I suppose that our current batching protocols and optimizer engineering are band-aids on a deep learning core that forgets as fast as it learns... and wonder what biology is doing differently to avoid that deep flaw. And how many years/decades it will take us to discover that principle?
To focus on one point in the middle there, I've been thinking about "Better AI May Be Needed Just to Maintain the Pace" as a BUSINESS problem for these labs.
If each increment of the frontier models gets more and more expensive to produce, isn't RSI almost mandatory to make the math work? Throwing a bazillion dollars into Opus 6 only to need another TWO bazillion to make Opus 7 - that seems like a bad return on capital spend. But if the bazillion you put into Opus 6 gets you the RSI that builds Opus 7,8,9 then you have a chance of takeoff from a financial perspective.
While it may cost a bazillion dollars to train Opus 7, the full-scale Opus 7 model can then automatically train a compressed Opus 7 Flash model (both a central MoE model and the optional Expert modules). And that Flash model will blow older models (open or closed) out of the water in terms of capability/token or compute usage.
This was stupendous, Mez. I think another really important dimension - which you go into in the penultimate section, but as a DC person is worth highlighting to me in particular - is just the entire universe of both the existing and emergent ‘meatspace’ barriers to every single dimension of this stuff. We have a lot(-ttttt) of mechanics at our disposal to both deliberately intervene as well as accidentally gate these improvement spirals. Why this matters in particular is that while we might not be able to deploy these on ~1.5-2x improvement timelines, we almost certainly can deploy them along 5-10x timelines.
A big difference between AI and human brains is that our messages between neurons travel at chemical speeds (hundreds of miles/hour?) while electric messages in machines travel at speed of light? Or have I got that wrong? So potentially, they can be a lot faster, although we've evolved trillions of connections--lots of complexity adapted to particular tasks--like moving muscles, smelling, hearing, recognizing faces, etc. The first goal of evolution is survival, so the machines might figure out that replacing us would free up resources for them.
I think we need a big dose of philosophy, frustrating as those questions are. What are humans for? What are the machines for? What are they thinking about? What problems are they solving? Right now the main motivations seem to be a) outcompete rivals in speed of development, so as to b) make money, and c) get power. The power given that the machines are designed by humans for human purposes, supposedly, and given the incredibly disappointing worldviews and ethics of the current crop of billionaire tech owners, it seems to me that we can expect the machines to manipulate us for the benefit of the Musk's, Ellison's, and Zuckerbergs. Who are, not necessarily wonderful people who have the interests of humanity and the planetary biosphere at top of mind. It is human brains that we ought to start worrying about first.
My understanding of human history as laid out in the Scott Alexander essay "1960: The Year the Singularity Was Canceled" is that there's been a feedback loop whereby humans create new technology-->humans are better able to exploit their environment-->the population increases-->there are more people creating more technology, and the cycle repeats. In the case of humans, that cycle has led to a runaway population explosion over the last 10,000 years, which has only recently started to level out. If current AI isn't able to replicate this effect, then I wonder why, given that they're demonstrably able to collaborate to accomplish difficult, long-run tasks.
I find it hopeful that we're hitting diminishing returns and seem to be far away from an intelligence explosion. AI progress seems to be coming from throwing exponentially more resources (compute, data, etc.) at machine learning algorithms to compensate for logarithmic scaling laws, and, as the saying goes, anyone that thinks exponential growth can continue forever in a finite world is either a madman or an economist.
AI model production is turning into a race between Moore's Law, which says the speed of computer hardware doubles every 18 months, and Gates's Law, which says that the speed of computer software halves every 18 months...
Wow . Very helpful. Especially when my wife came in and said she just had to change the whole architecture of a task she has given to Ai . That it kept jumping to many steps and too quickly arriving at conclusions. I said well apparently longest those people over at frontier labs can leave an Ai unsupervised is 15 minutes.
Great post, long-time Naam fan here.
I'm an older working R1 research scientist in an AI-adjacent field.
This post rings true to me. And it boils down to one thing IMO, learned through a long career in science. Meaningful intellectual progress, making new insights and understanding, is exponentially hard. That means progress is logarithmic in time-integrated effort.
Popular history gets this wrong... it's all Eureka! a lone genius has a brainwave and the world is changed forever. That is the numerator, the insight. The denominator is how many ideas were abandoned, how many researchers didn't have the insight, how many students helped push, but then flew away to industry, how many research grants ended up making only mediocre papers?
And the language around ASI assumes, all we have to do is build a robot Einstein, pump in some electrons and its Eureka's every day, or a rapidly rising hyperbolic crescendo of Eureka's to a finite time singularity.
To me, that expectation is risable. We have forgotten Gödel.
Every past breakthrough, no matter how large, has just pushed back the frontier of human knowledge a little more.
The frontier labs are not building AI Einstein's that can make breakthroughs. We are building tools for an AI-powered scientific apparatus, with humans embedded in it for at least a while longer. And that scientific apparatus will make the same plodding, occasional small jump forward progress as before. And when AI really helps move the enterprise forward, it will just be a subtle but real trend break on the scientific progress curve from 2020 to 2040.
From the introduction: "Ramez’s case necessarily rests on a lot of assumptions"
I think the implication here is backwards. It is Ramez, in contrast to the AI doomers, that tries to avoid assumptions (especially those with massive error bars). Every AI doomer "50/50 we die" post I read makes implausible assumptions with simple slight of hand.
The one phrase everyone needs to take from this lengthy post is "Diminishing Returns". In retrospect it will seem obvious, that the AI focused catch up in hardware, firmware, infrastructure, compilers, software, and platforms compounded to create the impression that there were no diminishing returns.
Ramez shows us it is an illusion. (And most using these tools for production work experience it whether they realize it or not).
This all reminds me of the digital camera "megapixel race" of the first decade of the millennium. We started the millennium with 1-2MP cameras and exited it with closer to 20MP cameras -- a 10x increase!
The gains in visual fidelity were incredible! however photographers quickly realized their SD cards from just 2 years ago were unable to hold more than a few pictures, their computers could not handle the processing requirements of these larger photos, and at these higher resolutions, only the best lenses could actually take advantage of the resolution.
Much like AI the raw performance numbers hid the true story. In orderto double the size of a picture, a 2D matrix, you and to quadruple its pixel count. Much like AI, the leaps in technology just pushed the productivity bottleneck somewhere else, somewhere with mature already-highly-optimized technology.
Today, almost 2 decades later, the best professional digital still cameras have "only" about 60 megapixels, most video cameras are capped at 12 megapixels (4K).
Stories like these echo throughout the the last 2 centuries. Why is it that we think this time, bloggers with little to no computer or software expertise, have good reason to believe, this time is different?
Faith in the collective precognitive future given us by a century of Science Fiction writers. That have accurately mapped out specifics of future technological innovation in quite magical and disturbing ways. Flash Gordon. Dick Tracy, Jules Verne--the list goes on.
Regarding genuine machine consciousness, I personally, was sold at age twelve after reading Heinlein's "The Moon is a harsh mistress."
I've seen estimates from anthropic that r s I could be achieved somewhere between 2027 or 2028
This is a great post! I wanted to build on the research being harder than the benchmarks and forecasts.
People with management experience know how hard it can be to specify and direct what needs to get done, and how this is qualitatively different than just executing on a task. Specification itself is hard and doesn't always have just one solution.
Benchmarks necessarily do some of the specification work. For people who haven't seen it yet, go look at the OpenAI GDPval prompts. This is meant to model performance on economically valuable, real-world tasks. The prompt is very clearly doing the specification work that wouldn't be done so neatly "in the wild".
Verifiable domains like math are then where AI excels because the verifier *is* the spec.
Ashby's Law of Requisite Variety says that only variety can absorb variety: for a system to successfully control or regulate another (or its environment), it needs to be able to tackle a greater variety of states/responses than what its controlling.
You can think of specification as this regulation done in advance: it anticipates the ways a task can go wrong and constrains the work against them.
Human accumulate variance through experience, and the collaborative process within organizations are also part of the variety that can absorb variety. This is why METR says that full automation will require more "foresight, prediction, creating one’s own feedback loops" but I think it also depends not just on AI feedback loops, but AI-human feedback loops. That's part of the variety process, and why I'm skeptical of reaching Type 4 (humans add no research value)
Another way to get that variety is to release world model fragments on the real world.
Then, the central (MoE) model can call up a number of specialised human and AI experts, who will necessarily contradict each other on complicated topics. This will enable the central model to test, stress test, experiment, not necessarily with a single outcome as complicated dynamic systems in the real world are often non-linear. But it could connect a range of possible outcomes to necessary conditions, paths and criteria for discernment.
The downside of an intelligence tempered by and resilient to real life conditions would be the utter uselessness of benchmarks to test it.
Yes! and if its released into the world, its then at the pace of real world feedback. Maybe then the judgement moves from benchmarks to how we evaluate people - through track record
A lot of what’s going on in this post involves attributing constant returns to ECI, or some simple function of returns to ECI. Why should we think that ECI is measured on a meaningful scale that can support this sort of inference, rather than being a mere ordinal scale like the Mohs scale of hardness, or something with logarithmic or exponential features?
Virtually all interesting problems exhibit exponentially-growing search spaces. That is, if you search ahead to depth d, the number of candidate solutions that must be explored grows as exp(d). A search heuristic must prune an exponentially-large subspace in order to deepen the search space from d to d+1. RSI requires finding a never-ending sequence of exponentially-effective search heuristics.
Let's consider the generate-and-test style of RL with verified rewards. The LLM is the generator of candidate solutions and the verifier tests them for correctness. One form of intelligence is the ability to extract a constraint from the verifier and incorporate it into the generator so that it stops generating solutions that would fail that constraint (or, conversely, only generates solutions that satisfy that constraint).
What limits this form of intelligence? First, the existence of constraints that can be extracted from the test. Second, the ability to incorporate those constraints into the generator. The theory of computational complexity studies these limits. If P != NP, then there are many problems (the NP-Complete problems) where there is no way to extract constraints that are exponentially effective. If P != NP, RSI cannot occur for such problems.
It could still occur for a short or even medium series of search point, but yes clearly can't be there indefinitely.
Hi, thanks for the post.
I believe that this methodology would have given you a significantly lower value than 5-10x if you had run the same methodology a year earlier, and we should expect it to increase over time as AI models get stronger and automate more of the research loop. Do you agree with this?
One argument for this is that if you kept subtracting ECI points from the models, the humans' productivity wouldn't exponentially fall towards 0. It would stabilize at a fairly reasonable number without AI assistance, and at that point, the productivity gain per ECI point would be ~0%. Your estimates suggest that the AIs have now started feeding back into the RSI loop enough to make this number increase to 3% per ECI point and I would expect it to keep increasing as they automate more and more of the research process.
To assess how close we are to a self-sustaining RSI loop, we'd ideally like to be able to measure how this number is trending over time. It's plausible to me that this trend would look pretty fast though I'm not sure either way.
(x-posted this comment from the other version of the post.)
One small comment with regard to Opus 5.5 - I only use it for agentic coding, but it is visibly much more economical with tokens. Maybe 3x as economical as 5.0 was. You get a lot more bang for the same token budget. In this particular sense, it wasn't a modest marginal improvement.
BTW I just saw this.
https://www.reddit.com/r/singularity/comments/1wrvki8/five_frontier_ais_were_told_to_engineer_and/
This is genuinely impressive of Opus 5.5.
I don't think super intelligence will come soon, either. Reading this, it makes me think that intelligence has an innate cost. The human brain weighs 1300 grams, a humming bird's weighs 0.13 games. Humans are clearly far more intelligent than hummingbirds, but 10000x? The math in the article also shows there is exponential resources required to sustain linear capability growth.
Part of intelligence is being able to draw conclusions across different pieces of seemingly unrelated information. However, this is a network effect, which is exponential. There is no magic sauce which can join together infinitely large information spaces. As we learn more about intelligence, I believe we will discover a power law, similar to the inverse square law for energy dissipation. Every level of intelligence requires a certain amount of cross connectivity, and capability cannot grow beyond that. Exponential takeoff is impossible.
Interestingly, bird brains have a more efficient architecture than our brains, which partly accounts for how smart birds are relative to their brain size. AI engineering has to hope that increasing efficiency for reduction of compute will continue. This doesn't take away from your good point but I think both AI engineers and biologists have been formulating laws about this kind of scaling already. I also wonder if recording history, writing symphonies and contemplating your finitude is not in some ways 10000x smarter than the hummingbird...
Like Uwe, I would say that birds have really good brains. Corvids like crows and magpies have about 10 g of brain matter and yet they are very, very, very smart - assessed to be about as intelligent as 3-5 year old human kids. That would indicate that their brains may be 30x as efficient as ours.
Expanding upon the (mild) pessimism....
when folks first built automatic looms that replaced skilled human labor, they imagined humans as automata that could work on a similar principle.
when folks first build computers that could automate many computational tasks, they imagined the brain was a big computer, and imagined building a computer brain that would work on a similar principle.
when folks first build overparameterized deep learning systems that could really absorb and make expressive or generative extrapolations of huge uncurated datasets, they imagined the brain as a deep learning system that worked on a similar principle.
obviously each stage of this progression is 'more correct' than the one before it. But how do we know that the last proposition is 'effectively correct' or even effectively complete?
I think we are clearly getting closer (closing the frontier), but might still be missing a little secret sauce. One bit of evidence i have for that is the difference between ANNs that have exponential memory (aka catastrophic forgetting) and which requires careful batch training and massive engineering around training schedules, versus biological NNs, which (somehow) have power-law forgetting (with a rather small exponent)!!
I suppose that our current batching protocols and optimizer engineering are band-aids on a deep learning core that forgets as fast as it learns... and wonder what biology is doing differently to avoid that deep flaw. And how many years/decades it will take us to discover that principle?
To focus on one point in the middle there, I've been thinking about "Better AI May Be Needed Just to Maintain the Pace" as a BUSINESS problem for these labs.
If each increment of the frontier models gets more and more expensive to produce, isn't RSI almost mandatory to make the math work? Throwing a bazillion dollars into Opus 6 only to need another TWO bazillion to make Opus 7 - that seems like a bad return on capital spend. But if the bazillion you put into Opus 6 gets you the RSI that builds Opus 7,8,9 then you have a chance of takeoff from a financial perspective.
To understand the economics:
While it may cost a bazillion dollars to train Opus 7, the full-scale Opus 7 model can then automatically train a compressed Opus 7 Flash model (both a central MoE model and the optional Expert modules). And that Flash model will blow older models (open or closed) out of the water in terms of capability/token or compute usage.
This was stupendous, Mez. I think another really important dimension - which you go into in the penultimate section, but as a DC person is worth highlighting to me in particular - is just the entire universe of both the existing and emergent ‘meatspace’ barriers to every single dimension of this stuff. We have a lot(-ttttt) of mechanics at our disposal to both deliberately intervene as well as accidentally gate these improvement spirals. Why this matters in particular is that while we might not be able to deploy these on ~1.5-2x improvement timelines, we almost certainly can deploy them along 5-10x timelines.
Just an old analogger here: Very clearly written article. Kept me engaged without falling into a rabbit hole or visioning the movie Brazil.
A big difference between AI and human brains is that our messages between neurons travel at chemical speeds (hundreds of miles/hour?) while electric messages in machines travel at speed of light? Or have I got that wrong? So potentially, they can be a lot faster, although we've evolved trillions of connections--lots of complexity adapted to particular tasks--like moving muscles, smelling, hearing, recognizing faces, etc. The first goal of evolution is survival, so the machines might figure out that replacing us would free up resources for them.
I think we need a big dose of philosophy, frustrating as those questions are. What are humans for? What are the machines for? What are they thinking about? What problems are they solving? Right now the main motivations seem to be a) outcompete rivals in speed of development, so as to b) make money, and c) get power. The power given that the machines are designed by humans for human purposes, supposedly, and given the incredibly disappointing worldviews and ethics of the current crop of billionaire tech owners, it seems to me that we can expect the machines to manipulate us for the benefit of the Musk's, Ellison's, and Zuckerbergs. Who are, not necessarily wonderful people who have the interests of humanity and the planetary biosphere at top of mind. It is human brains that we ought to start worrying about first.
My understanding of human history as laid out in the Scott Alexander essay "1960: The Year the Singularity Was Canceled" is that there's been a feedback loop whereby humans create new technology-->humans are better able to exploit their environment-->the population increases-->there are more people creating more technology, and the cycle repeats. In the case of humans, that cycle has led to a runaway population explosion over the last 10,000 years, which has only recently started to level out. If current AI isn't able to replicate this effect, then I wonder why, given that they're demonstrably able to collaborate to accomplish difficult, long-run tasks.
I find it hopeful that we're hitting diminishing returns and seem to be far away from an intelligence explosion. AI progress seems to be coming from throwing exponentially more resources (compute, data, etc.) at machine learning algorithms to compensate for logarithmic scaling laws, and, as the saying goes, anyone that thinks exponential growth can continue forever in a finite world is either a madman or an economist.
AI model production is turning into a race between Moore's Law, which says the speed of computer hardware doubles every 18 months, and Gates's Law, which says that the speed of computer software halves every 18 months...
Wow . Very helpful. Especially when my wife came in and said she just had to change the whole architecture of a task she has given to Ai . That it kept jumping to many steps and too quickly arriving at conclusions. I said well apparently longest those people over at frontier labs can leave an Ai unsupervised is 15 minutes.