The Three Filters: Why Almost Every Plan to Survive ASI Fails Miserably

This post is based on my personal views, which mostly overlap with the views of my employer Control AI but does not necessarily fully reflect them. This applies in particular, but not exclusively, to technical opinions about AI development and geopolitical predictions.You might’ve heard that superintelligent AI (ASI) poses extreme risks like human extinction and other comparably undesirable outcomes.If you’re like me, you probably looked into possible solutions. And if so, you may have found a range of reassuringly tractable theories of change. To name a few:Technical AI safety research agendasRacing to ASI so your favorite company or country can get there first and prevent anyone else from building “bad” ASIBuilding a good ASI and handing it control over the whole world (so that we don’t have to be subject to any evil human dictators)If you think about it, all of these feel quite convenient, especially if you’re a tech-leaning person: you don’t need to change your career at all. Just keep working on your favorite ASI project, and things will work out.It’s quite easy to come across theories that predict good outcomes without needing to change your strategy at all, even if you’re actively working to bring about ASI as soon as possible. I see these as being mostly semantic stopsigns. Most of them around about AI alignment being feasible:AI alignment is easy and people are working hard on it, so it’ll probably be ok.AI will help us do alignment research.Iterative deployment will help us catch problems before AI gets too powerful.In this post, I want to show you that even if the theories of change mentioned above were applied extremely successfully or if AI alignment actually turned out to be technically easy, all the value in the world is still on track to be destroyed because of AI development. This means, mostly, human extinction. It also includes scenarios that don’t literally qualify as human extinction but are still comparably undesirable. For example, the least-ba