Alignment & Succession: The Ideology of Successionism

(Originally published on No Set Gauge.)Gustave Moreau, The Frogs Asking For A King In the course of building a better world, people ask each other many questions. Which things should be managed by the government and which left to the market? What sort of technology, if any, is so dangerous that it should be kept secret, access curtailed, or development avoided? Is goodness fundamentally about following the right rules, achieving the right outcomes, or having the right character?Reasonable people have different opinions on all these questions. But recently, Silicon Valley has seen lively debate on a question you’d hope was all too obvious: should humanity continue existing?The idea that it shouldn’t was named successionism by Andrew Critch, and is motivated by the speed and power of AI development.Some examples:Already back in 2013, Elon Musk, freaked out by Demis Hassabis’s warnings about AI risk, got into an argument with Larry Page about whether it matters if AI replaces humanity. Page called it just the next stage of evolution and those that resist it “speciesists”. Elon, who has often had good instincts on goals but is not known for his eloquence, retorted “Well, yes, I am pro human, I fucking like humanity, dude”. Walter Isaacson, in his Elon biography, portrays this incident as one of the catalysts of OpenAI starting.Richard Sutton, Turing award winner and reinforcement learning pioneer, tells us we should “prepare for, but not fear, the inevitable succession from humanity to AI“ (see link for a full talk), and approvingly quotes Moravec (of Moravec’s paradoxfame) writing in 1998 “We can probably arrange for a comfortable retirement before we fade away”. Recently Sutton has elaborated on his thoughts in a Dwarkesh interview.Podcaster, blogger, and AI market research company founder Daniel Faggella says“that the great (and ultimately, only) moral aim of AGI should be the creation of Worthy Successor – an entity with more capability, intelligence, ability to surv