And what happens next?

In the game "The choice before us" by Nick Shapiro,[1] you are put in the shoes of an AI company leader. You grow your business. You unlock "wonders", such as curing cancer. All the while, you're attempting to avoid your product getting smart enough to escape and take over. You win by achieving 5 wonders without unleashing uncontrolled AI.I love this game, but it has the major flaw that when you win, you are normally very close to superintelligence. What happens afterwards? You turn the GPUs off? Go home? Get some sleep? The game seems to think so. This failure to ask "What happens next?" seems to be a broader phenomenon within the AI community. It was in fact the sole question I needed to ask a capabilities researcher for them to take the threat of superintelligence seriously. It's my main weapon against people claiming there are many possible worlds "where only 90% of people die" (if a rogue AI has gone off the rails and killed 90% of your population, you probably no longer have control of the planet, and I have little faith in the survival of everybody else). More broadly, I just wish people would ask this question more often."But Sean!" you say. "I cannot keep asking what comes next forever. I'll end up wondering what happens after the pope becomes US president in the year 2124." And you would be correct! You cannot, in fact, keep on doing this forever. The tree of possibilities is infinite, and spending your life exploring them is reserved for the brave of heart, the noble of mind, and those who have nothing more productive to be doing. We would be better off finding a place to stop. To work out how, let's look at a simpler version. In particular, the game of chess. More specifically, this position:In case you're wondering, this position is a messFor the uninitiated, this is a complex position. The white king is threatened. The black queen is under attack. So is the white rook, a white bishop and one of the white knights. More broadly, the arrows in the image s