Superintelligence of the gaps
Many classic AI doom scenarios rely on superintelligence using its vastly superior intelligence to outplan, outcompete and outkill you. I partly believe this: superintelligence would definitely outkill me. But I don't believe we will build such superintelligence; not because humans are the apex of intelligence, but because superintelligence, implicitly, has always been about a gap: the gap between the current best intelligence and the newly created one. We're not in the world where AIs are being created with large gaps of intelligence between each other. Rather, we are in an iterative intelligence development and deployment world. It is technically easy to not have large gaps of capabilities between the current best model and the next, it is ~easy (if costly) to evaluate at regular checkpoints, and ~continuous deployment allows there to be no large gap in deployment either. We can thus steer away from a large number of doom scenarios (those where new AI uses its greater capabilities to take over) by simply not creating&deploying models much smarter than the previous thing. The current most intelligent and aligned beings should always be supervising their successor, using more total resources at first, such that they can't effectively be tricked/subverted. I guess the above is something many "AI optimists" have in mind and I don't think the technical ease of avoiding large capabilities gaps should be much of a crux. Whether in practice we'll be avoiding these gaps seems the more interesting crux for "fast misaligned AI takeover" scenario discussion. This is correctly done in @Daniel Kokotajlo et al's AI 2027: the bad ending is caused by pressures to premature deployment leading to using a suspected misaligned system, not by technical impossibility of knowing it's misaligned. It is also what makes that particular scenario unlikely to happen. The leading companies will be more careful than that if they had that level of evidence of misalignment in powerful systems. (I