My thoughts on AI safety

Contemplating the unknowable.

Art by Nano Banana

Today at a Christmas party I had an interesting and productive discussion about AI safety. I almost can’t believe I just typed those words — having an interesting and productive discussion about AI safety is something I never expected to do. It’s not just that I don’t work in AI myself — it’s that the big question of “What happens if we invent a superintelligent godlike AI?” seems, at first blush, to be utterly unknowable. It’s like if ants sat around five million years ago asking what humans — who didn’t even exist at that point — might do to their anthills in 2025.

Essentially every conversation I’ve heard on this topic involves people who think about AI safety all day wringing their hands and saying some variant of “OMG, but superintelligent AI will be so SMART, what if it KILLS US ALL?”. It’s not that I think those people are silly; it’s just that I don’t feel like I have a lot to add to that discussion. Yes, it’s conceivable that a super-smart AI might kill us all. I’ve seen the Terminator movies. I don’t know any laws of the Universe that prove this won’t happen.

One response you can have to this is to conclude that because superintelligent AI might kill us all, that we should make very, very sure that no one ever invents it. This is the thesis of If Anyone Builds It, Everyone Dies, by Eliezer Yudkowsky and Nate Soares:

This sort of reminds me of Gregory Benford’s famous admonition to “Never do anything for the first time.” You can make arguments for unforeseen catastrophic consequences for lots of different technologies. Nuclear weapons might kill us all of course, but so might synthetic biology or genetic engineering. Social media or other media technologies might collapse our societies. Video games might entrance us into virtual lives so that we never reproduce. In fact, it’s quite possible that universal literacy and reduced child mortality have already condemned humanity to dwindle comfortably into nothingness, with no artificial superintelligence required.

Personally, I do think that being so afraid of existential risk that you never invent new technologies is probably a suboptimal way for an intelligent species like ours to spend our time in this Universe. It’s certainly a boring way for us to spend our time; imagine if we had been so afraid that agriculture would kill us that we remained hunter-gatherers forever? My instinct says we should see how far technology can take us, instead of choosing to stagnate and remain mere animals.

But yes, OK, a superintelligent techno-god might kill us all. We can’t really know what it would want to do, or what it might be capable of doing. So if you really really want to be absolutely sure that no superintelligent techno-god will ever kill us all, then your best bet is probably to just arrest and imprison anyone who tries to make anything even remotely resembling a superintelligent techno-god. This is the approach of the Turing Cops in Neuromancer, the most famous cyberpunk novel.

And yet today at the Christmas party, against my better judgment, some people who think about AI safety for a living roped me into a serious discussion about their field of research. And to my utter astonishment, they actually thought I had some novel and interesting things to say.

If I can impress AI safety researchers with my thoughts on AI safety, I might as well write them up in a blog post. First, I’ll explain why I’m not very afraid that superintelligent AI will destroy humanity, and suggest some things I think we could do to minimize the risk that it will. And then I’ll explain what actually scares me about AI.

Stoner kids, Singapore, and French forests