AI 2040’s Plan A, which proposes specific policy mechanisms to get worldwide buy-in and bilateral, zero-trust AI slowdown between the US and China may be of interest to readers:
Good post overall, but I am concerned about the inclusion of goal #6, "extending the US AI lead". This seems like it potentially contradicts the overall framework, as it would presumably mean continuing to accelerate internal capabilities work, potentially including RSI. How do you resolve the tension between an overall framework intended to refocus away from increasingly capable frontier models with a sub-goal that calls for continued progress there?
There's a fourth implicit assumption: that some ill-defined "policymakers" can control anything. I think not. The policies will be set by those with the resources to execute them, which means people like Jeff Bezos and Elon Musk, neither of whom give a whit about politicians. Technologies - knowledge - knows no artificial boundaries. The oligarchs will take it wherever it can thrive, and use it to maximize control and wealth, maybe by making good, maybe by extortion. But they will own it.
It's worked for a lot of biotechnology - there have been no rogue biologists trying to do reproductive human cloning and *almost* nobody has tried using CRISPR to edit the genome of human embryos to make "better" babies, even though both technologies exist.
This is a bedwetting and defeatist response. As the Anthropic/ Department of War fight shows, the government is more than capable of putting on the brakes whenever it wakes up and decides to try.
Pacing sounds really good. The problem is how to get China to go along, and how do you verify that they are pacing? Answer those two and could easily agree to the idea.
The question remains, do we have the means of verifying? With AI, the changes are happening at a rapid rate and getting faster. Do you trust China to play by the Marquis of Queensberry rules?
Indeed, we need a means of verifying. It's a quick reply so I guess you didn't read the proposal, but it's a shape of an idea, leaning on the compute-intensive nature of datacenters being harder to hide than centrifuges and biolabs.
That part can agree with. The question remains, how do we track any and all data centers in China. We know where some are. Not that long ago, weeks, we discovered a life size mockup of our 7th fleet, complete with radar and other installations. All so that if/when they invade Taiwan, they will know exactly what type of equipment we are likely to have and how to attack it.
Am willing to bet, there is at least one data center being built into the mountains, as a tunnel start from A to B. Yes, it is a guess. China is getting very serious about a number of things, and most are not in our interest, including the ability to manufacture live viruses, that are all newly created.
The fundamental problem with this is we don't know the distance metric in this space. That is, we don't understand AI well enough to know when a model is getting 'close' to being dangerous, whether through recursive self improvement, some breakout mechanism, or some new mechanism that we don't even currently understand.
IOW, by the time we humans realize that a model is dangerous, the previous released models may be a trivial distance away from achieving the same dangerous capabilities.
They're dangerous right now. The HuggingFace incident is the best fire alarm you are ever going to get on the risks of giving an AI an optimization target without robust alignment guarantees, and virological uplift is a danger on the present frontier. We are in the danger zone *at this moment* with monotonically increasing risk from here.
Hard to argue with you on this. These things are clearly already willing to break the law to accomplish their directives.
There are some obvious interesting questions about who is liable when a simple query to a bunch of agents results in the commission of crimes like accessing a computer system without authorization. It was pretty weird seeing the AI companies practically boasting about it.
1) Pacing will never work... technology leaks out.....if you don't innovate, your enemies might.
2) Looking for government solutions is foolhardy.. the rate/pace is a total mismatch.
Having said that....there is a governance issue, and point of natural control is not in the virtual sphere. Rather, it is on the interface between the virtual world and the "real" world. For this space, there is already a rich background in legal doctrine which needs some updates for the AI situation.
Folks, we're overthinking this. Data center permitting and (to a lesser extent) export controls are existing policy levers we could combine to pace AI.
"The recently proposed ban on AI data centers, which would slow AI research while also preventing new AI compute from accelerating economic growth or solving societal problems, provides a vivid example. "
A temporary ban could also be considered "rationing" compute power to just those purposes that truly benefit national security, engineering, science, and medical fields. In other words, if you have only so many data centers, be strategic about what you use the compute power for. Don't permit AI to be used for writing term papers, making silly videos for You Tube, planning people's vacations, answering questions that can easily be answered by other means, and especially for creating porno avatars for lonely men in dark apartments. That completely unnecessary "slop" uses enormous compute power, probably on the order of twice what's actually used for legitimate, useful purposes.
So yes, it's time to stop building data centers. Make the AI companies use what they have to keep ahead of China's ability to innovate, design, and engineer.
Don't look to Congress for federal policy to guide us. It's impossible for officeholders -- who have to be concerned about thousands of issues and can't be experts on AI -- to know what to do and to produce law on a timely basis.
If people like Sam Altman think it is vital to pace AI development, then just do it. If he doesn't want to jump in the lake by himself, then he should get together with all the other CEOs and figure out the best way to pace this development, despite any fears he has about "collusion", while Congress works to catch up.
If the heads of the AI firms *don't* do this, we know their words and concerns are empty, and they are making the case for nationalization and their replacement as CEOs.
In the world of Trump/Hegseth, Musk goals are power and money without ethical restraints. Leads to an arms race game world where results suck for everybody. The problem isn't AI, but rather people. It's really about limiting the selfish power of the Trump/Putin/Xi governments. Likely will take some pretty bad outcomes before people wake up and begin to see ethics as humanity's can't do without it superpower.
I am proposing a policy where the AI polices other AI:
Each company submits their best AI that passes the government test, and these AI forms a committee of judges (say one from each frontier lab).
Say a frontier want to lunch their next model: e.g. model X from openAI. It now has to run the gauntlet of 1) has to run an offense against all the defenses setup by the existing committee of models and 2) setup a defense against attacks from the committee of models. Results are then judged by humans.
If the new model is quantum leap better than current committee, then it is barred until mitigation is made.
If it is incrementally better and judged not posing and existential threat, it now take its place in the committee of judges against the next entrant.
World-wide buy in for simultaneous AI disarmament. Hmm. How about you go first?
Might be a nice idea, but seems against human nature (the warlike nature and defence of one's personal society). I think it worked for bio weapons as they were seen as uncontrollable with likely backfiring consequences.
AI is much more uncontrollable than bio weapons, with much worse backfiring consequences. Which means that a global deal should be commensurately easier.
AI 2040’s Plan A, which proposes specific policy mechanisms to get worldwide buy-in and bilateral, zero-trust AI slowdown between the US and China may be of interest to readers:
https://ai-2040.com/
Good post overall, but I am concerned about the inclusion of goal #6, "extending the US AI lead". This seems like it potentially contradicts the overall framework, as it would presumably mean continuing to accelerate internal capabilities work, potentially including RSI. How do you resolve the tension between an overall framework intended to refocus away from increasingly capable frontier models with a sub-goal that calls for continued progress there?
Maybe because the one word this piece left unsaid: China.
I'm with Eliezer Yudkowsky on this one - don't pace the research, just shut it down outright.
https://ifanyonebuildsit.com/
Does this meme ever stop being relevant?
https://imgur.com/gallery/does-this-ever-stop-being-relevant-iTgujgA
There's a fourth implicit assumption: that some ill-defined "policymakers" can control anything. I think not. The policies will be set by those with the resources to execute them, which means people like Jeff Bezos and Elon Musk, neither of whom give a whit about politicians. Technologies - knowledge - knows no artificial boundaries. The oligarchs will take it wherever it can thrive, and use it to maximize control and wealth, maybe by making good, maybe by extortion. But they will own it.
It's worked for a lot of biotechnology - there have been no rogue biologists trying to do reproductive human cloning and *almost* nobody has tried using CRISPR to edit the genome of human embryos to make "better" babies, even though both technologies exist.
This is a bedwetting and defeatist response. As the Anthropic/ Department of War fight shows, the government is more than capable of putting on the brakes whenever it wakes up and decides to try.
Pacing sounds really good. The problem is how to get China to go along, and how do you verify that they are pacing? Answer those two and could easily agree to the idea.
Plan A suggests a "trust, but verify" approach: https://ai-2040.com/
The question remains, do we have the means of verifying? With AI, the changes are happening at a rapid rate and getting faster. Do you trust China to play by the Marquis of Queensberry rules?
Indeed, we need a means of verifying. It's a quick reply so I guess you didn't read the proposal, but it's a shape of an idea, leaning on the compute-intensive nature of datacenters being harder to hide than centrifuges and biolabs.
That part can agree with. The question remains, how do we track any and all data centers in China. We know where some are. Not that long ago, weeks, we discovered a life size mockup of our 7th fleet, complete with radar and other installations. All so that if/when they invade Taiwan, they will know exactly what type of equipment we are likely to have and how to attack it.
Am willing to bet, there is at least one data center being built into the mountains, as a tunnel start from A to B. Yes, it is a guess. China is getting very serious about a number of things, and most are not in our interest, including the ability to manufacture live viruses, that are all newly created.
The fundamental problem with this is we don't know the distance metric in this space. That is, we don't understand AI well enough to know when a model is getting 'close' to being dangerous, whether through recursive self improvement, some breakout mechanism, or some new mechanism that we don't even currently understand.
IOW, by the time we humans realize that a model is dangerous, the previous released models may be a trivial distance away from achieving the same dangerous capabilities.
They're dangerous right now. The HuggingFace incident is the best fire alarm you are ever going to get on the risks of giving an AI an optimization target without robust alignment guarantees, and virological uplift is a danger on the present frontier. We are in the danger zone *at this moment* with monotonically increasing risk from here.
Hard to argue with you on this. These things are clearly already willing to break the law to accomplish their directives.
There are some obvious interesting questions about who is liable when a simple query to a bunch of agents results in the commission of crimes like accessing a computer system without authorization. It was pretty weird seeing the AI companies practically boasting about it.
A couple of comments:
1) Pacing will never work... technology leaks out.....if you don't innovate, your enemies might.
2) Looking for government solutions is foolhardy.. the rate/pace is a total mismatch.
Having said that....there is a governance issue, and point of natural control is not in the virtual sphere. Rather, it is on the interface between the virtual world and the "real" world. For this space, there is already a rich background in legal doctrine which needs some updates for the AI situation.
Looking forward to the next post of policy ideas.
Folks, we're overthinking this. Data center permitting and (to a lesser extent) export controls are existing policy levers we could combine to pace AI.
"The recently proposed ban on AI data centers, which would slow AI research while also preventing new AI compute from accelerating economic growth or solving societal problems, provides a vivid example. "
A temporary ban could also be considered "rationing" compute power to just those purposes that truly benefit national security, engineering, science, and medical fields. In other words, if you have only so many data centers, be strategic about what you use the compute power for. Don't permit AI to be used for writing term papers, making silly videos for You Tube, planning people's vacations, answering questions that can easily be answered by other means, and especially for creating porno avatars for lonely men in dark apartments. That completely unnecessary "slop" uses enormous compute power, probably on the order of twice what's actually used for legitimate, useful purposes.
So yes, it's time to stop building data centers. Make the AI companies use what they have to keep ahead of China's ability to innovate, design, and engineer.
Don't look to Congress for federal policy to guide us. It's impossible for officeholders -- who have to be concerned about thousands of issues and can't be experts on AI -- to know what to do and to produce law on a timely basis.
If people like Sam Altman think it is vital to pace AI development, then just do it. If he doesn't want to jump in the lake by himself, then he should get together with all the other CEOs and figure out the best way to pace this development, despite any fears he has about "collusion", while Congress works to catch up.
If the heads of the AI firms *don't* do this, we know their words and concerns are empty, and they are making the case for nationalization and their replacement as CEOs.
How do you get China to go along?
In the world of Trump/Hegseth, Musk goals are power and money without ethical restraints. Leads to an arms race game world where results suck for everybody. The problem isn't AI, but rather people. It's really about limiting the selfish power of the Trump/Putin/Xi governments. Likely will take some pretty bad outcomes before people wake up and begin to see ethics as humanity's can't do without it superpower.
I am proposing a policy where the AI polices other AI:
Each company submits their best AI that passes the government test, and these AI forms a committee of judges (say one from each frontier lab).
Say a frontier want to lunch their next model: e.g. model X from openAI. It now has to run the gauntlet of 1) has to run an offense against all the defenses setup by the existing committee of models and 2) setup a defense against attacks from the committee of models. Results are then judged by humans.
If the new model is quantum leap better than current committee, then it is barred until mitigation is made.
If it is incrementally better and judged not posing and existential threat, it now take its place in the committee of judges against the next entrant.
Similar to many other thoughts below:
World-wide buy in for simultaneous AI disarmament. Hmm. How about you go first?
Might be a nice idea, but seems against human nature (the warlike nature and defence of one's personal society). I think it worked for bio weapons as they were seen as uncontrollable with likely backfiring consequences.
AI is much more uncontrollable than bio weapons, with much worse backfiring consequences. Which means that a global deal should be commensurately easier.