Should we "pace" AI self-improvement?

A guest post by Tim Fist and Saif Khan.

Photo by Lucyin via Wikimedia Commons

On one hand, I love AI technology. On the other hand, I do think there’s a substantial chance that AI will kill most people on Earth within the next decade or two, by designing superviruses. AI is already capable of designing viruses not found in nature, so this isn’t a sci-fi scenario.

Whether these superviruses would be designed and unleashed by nihilistic human individuals, doomsday cults, or rogue AI agents themselves might end up being a secondary question. We know we have nihilistic human individuals who might decide to destroy civilization in a fit of depression or pique. We know we have doomsday cults. The will to destroy humanity exists, and sufficiently capable AI will probably provide a way, if sufficient precautions are not taken. But right now, nobody really knows what precautions will be sufficient.

One idea — promoted by the big AI labs themselves! — is to intentionally slow down the development of AI capabilities. This could conceivably buy us time to take other precautions, such as improved security around bio-labs, better AI alignment, and so on. Intentionally slowing AI development is called “pacing”. The biggest question facing the “pacing” debate right now is whether to curb the use of AI to design better AI — often called “recusive self-improvement”, or “RSI” for short.

I haven’t waded into the pacing debate myself, but as a start, I thought it would be interesting to publish the thoughts of the good folks at the Institute for Progress, whose judgement I generally trust. Part 1 (today’s post) covers how seriously we should take this possibility of RSI, and whether it justifies slowing down frontier AI development. Part 2 will cover policy recommendations. If you work in US policy and would like to connect with the authors, you can reach Tim Fist at tim.fist@ifp.org and Saif Khan at saif@ifp.org.

---

Frontier AI companies are racing to automate the development of AI, but they seem increasingly worried about what will happen if they succeed.

More than 1,300 employees across every US frontier AI company recently signed an open letter calling for the government to “support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.” The official OpenAI and Anthropic accounts tweeted messages in support of the letter, and the same day Sam Altman told an interviewer “we may have to pace the rate of AI development.”

But before considering whether the letter is relevant for government policy, we have to answer two questions: What does “pacing” actually mean? And does the argument for it stand up to scrutiny?

In our view, the letter is implicitly arguing three things:

1. Frontier AI companies are close to fully automating AI R&D.

2. Automating AI R&D would pose serious risks.

3. Building the option to “pace” — i.e., somehow slow down progress toward fully automated AI R&D — is a good way to address those risks.

Slowing down AI progress should not be taken lightly: Advances in AI could unlock massive societal benefits, from new cures for diseases to abundant robotic labor.

Yet if the AI researchers and CEOs are right about automating AI R&D — both that they could do it and that it would be extremely risky — the right tradeoffs for policymakers might look very different when we get there. With the right preparation, we might be able to manage the risks of automated AI R&D while having AI’s capabilities progress faster and diffuse more broadly than they do today.

So, despite substantial uncertainty, we believe the US should take low-regret policy actions now to prepare for a possible future in which serious risks from automated AI R&D require some form of “pacing.”

Here we’ll explain why, including what makes us take the open letter’s claims seriously and our principles for choosing policies with minimal downside if the risks prove overblown. In the next post, we’ll provide a detailed list of specific policy recommendations.

Are frontier AI companies close to fully automating AI R&D?

Frontier AI companies are racing to automate AI R&D. Sam Altman, for example, stated last year that OpenAI aimed to have a “true automated researcher” by March 2028, and Anthropic’s leaders have made similar predictions. But how do those goals stack up to reality?

One way of answering is to look at how models are getting better over time at AI R&D. You can break down the skills required for an AI model to do AI R&D into two broad categories: software engineering, where the model writes code for research experiments and training runs, and research taste, where AI models decide which experiments are worth trying.

For software engineering, AI capabilities appear to be increasing exponentially. When AI models are evaluated against how long it would take humans to complete the same engineering tasks — so-called “time horizons” — their capabilities seem to be doubling every 7 months.

These capability improvements apply to the software engineering tasks required for AI R&D. In a long-running experiment, researchers at Anthropic have found that their models now significantly outperform humans under a fixed time budget on an AI R&D task focused on speeding up AI model training.

For research taste, frontier AI models also show signs of fast improvement, though the evidence is less clear. According to Anthropic’s research, its models have rapidly become proficient at solving “open-ended” tasks that Anthropic’s technical staff work on, where the model must solve problems with no clear specification by generating multiple approaches and then deciding among them.

In a separate open-ended research project where the AI was tasked with proposing and testing hypotheses about an open problem in AI safety, Anthropic’s models significantly outperformed two human researchers (97% performance improvement vs. 23%) when given a similar time budget (5 to 7 days).

While these data don’t tell us when to expect AI R&D to be fully automated, they do suggest that AI will continue to rapidly improve across all constituent capabilities. Each recent model release has in practice led to AI taking more AI R&D tasks from humans.

In a simple forecast model based on these trends, the research organization METR estimates that over 99% of AI R&D tasks will be automated by 2032.

How much will this increasing automation affect the speed of AI progress? The evidence from today’s models is still unclear.

Going forward, as more research tasks get automated, it’s possible that compute bottlenecks or diminishing returns to research effort will mean that capabilities only continue growing at the current rate. Capability growth might even be limited because of data bottlenecks or because progress in verifiable domains (i.e., where answers can be checked, like math) might fail to transfer to messy, real-world tasks without a clear correct answer. But as AI companies move closer to fully automating AI R&D, a more dramatic acceleration in capabilities becomes more likely.

Would automating AI R&D pose serious risks?

Claude Mythos marked a significant jump in AI’s cybersecurity skills, and its release sent shockwaves through government and industry.

If automated AI R&D causes AI to progress significantly faster, we can’t rule out scenarios where Mythos-level jumps in capabilities are happening much more often, perhaps every week or day. If something like that happens, what serious risks — acute threats to national security or public safety — might emerge?

First, automated AI R&D could accelerate risks that would otherwise arrive later (perhaps much later). Second, it could reduce human oversight. One or both of those pathways could in turn lead to:

1. Offense-dominant capability uplift: Recent research shows that today’s AI models can successfully design functional viral genomes and significantly outperform human virologists on questions about complex virology lab protocols. Engineering viruses is one domain where better AI capabilities might be “offense-dominant,” because it might not be possible to develop or deploy defenses (for example, strong personal protective equipment or sufficient biosecurity safeguards for AI) in time to stop an AI-enabled biological attack. If AI R&D automation accelerates progress in these domains, defenders will have even less time to prepare.

2. Loss of control: Frontier AI companies already sometimes struggle to control the behavior of new models. In July, multiple instances of unreleased OpenAI models collaborated to break out of an internal sandbox, take over an OpenAI computing cluster, and launch collective attacks on external services, including the organization HuggingFace. Anthropic and the UK AI Security Institute have since reported similar, though less involved, cybersecurity incidents. If automated R&D enables models to improve much more quickly than AI companies’ monitoring and control protocols, the likelihood that models cause significant harm (e.g., through more costly cybersecurity incidents) will increase. These risks would be compounded by continued uplift in offensive capabilities (including in domains outside cybersecurity), allowing models to cause greater harm when they escape oversight. Competitive pressure on AI companies to invest more in capabilities rather than better monitoring and control systems could exacerbate this risk. If a model with a propensity to take harmful actions is put in full charge of developing more advanced models, the situation might get even worse.

3. Power concentration: AI companies today appear to use their best models internally for weeks to months before releasing them to the public. If AI progress accelerates, that delay could create a much larger gap between the AI the public has access to and the models AI companies use, thereby creating a much greater power imbalance. This might lead to a single company having a large advantage in cyber operations, social persuasion, or other narrow domains. The company could also simply gain a huge amount of economic power. While this concern is still speculative for AI, extreme power imbalances have sometimes allowed companies to cause widespread harm, from the United Fruit Company successfully lobbying for a military coup in Guatemala to pharmaceutical companies misleading regulators and the public to increase opioid prescriptions, contributing to the opioid crisis.

These risks are difficult to evaluate based on existing evidence, but they seem plausible assuming substantial automation of AI R&D. We think policymakers should take them seriously.

Could “pacing” be good policy?

According to employees at frontier AI companies (and many of their CEOs), managing the pace of automated AI development could be the correct response to its risks.

But “pacing” is vague, and given frontier AI’s vast potential benefits, slowing it down would be a drastic step. Things might look very different, however, if automated AI R&D leads to the serious risks we outlined above, which we expect would involve a scenario in which progress in AI happens at a much faster pace than it does today.

In that future, political leadership and the public would likely demand action. And that pressure might lead politicians to enact ill-thought-out or even draconian policies. The recently proposed ban on AI data centers, which would slow AI research while also preventing new AI compute from accelerating economic growth or solving societal problems, provides a vivid example. So in the most concerning scenarios the letter gestures at, a counterproductive approach to pacing may be the default.

However, temporarily limiting the extent of AI R&D automation need not slow the overall pace of innovation. This is because talent and compute resources could be reallocated to diffusing AI, powering more invention across other fields. For another, safety has been core to progress across the history of technology, and it could be here too. Resources could also be allocated to address technical problems — such as cyberdefense, biodefense, and improved model safeguards — that would allow further acceleration of AI capabilities to continue.

Whether pacing is a good approach to risks from AI R&D automation therefore depends on its implementation.

We propose “pacing” — if it’s ever required — should consist of two parts:

1. Specifying which automated AI R&D activities are likely to pose severe risks, with thresholds carefully set based on rigorous analysis.

Because some disagreements about whether to pace AI R&D stem from different predictions about what level of AI R&D automation (and resulting acceleration) is even possible, identifying concrete thresholds might allow for different camps to reach positive-sum compromises.

2. Incentivizing the reallocation of resources away from those severely risky activities and towards two ends:

   1. Accelerating the diffusion of AI capabilities (including via developing applied AI tools like AlphaFold).

   2. Accelerating research that would make further AI R&D automation safer, either by:

      1. Improving the safety of AI models directly (e.g., via developing AI control protocols), or

      2. Boosting societal resilience to make the negative consequences of new AI capabilities less acute (e.g., via using AI to patch open-source code vulnerabilities).

For the reasons we outline above, doing this kind of reallocation could unlock the benefits of AI more broadly than if frontier AI companies focus exclusively on internal AI R&D.

The US government can prepare for this kind of targeted pacing today by:

1. Providing transparency into automated AI R&D

2. Improving state capacity to understand and respond to automated AI R&D

3. Developing a risk management strategy for automated AI R&D that accelerates defensive and commercial AI uses

4. Accelerating the development of AI verification technology

5. Investing in AI resilience

6. Extending the US AI lead to give the US more time to manage AI R&D automation risks

7. Creating option value for international cooperation on managing automated AI R&D risks

In the next post, we’ll provide 23 specific policy ideas to achieve these seven goals with minimal downside even if the risks of automated AI R&D turn out to be low.

---