Why I think a global AI pause (almost) certainly won't happen
Nuclear arms treaties happened AFTER nukes had been demonstrated. AI pause would need to happen BEFORE ASI comes into existence.If I had to explain the issue in just 2 sentences, those are the sentences I would say. Now let's elaborate:1) Explaining the danger of nukes is really easy. "A giant explosion will evaporate an entire city or even a small country". That's it, congratulations, you have successfully explained the danger of nukes. Now try to explain the danger of superintelligent AI in a way that will be convincing to the average person AND to policymakers, and won't require a 2-hour long lecture with a PowerPoint presentation.ASI risk is abstract and cannot be easily demonstrated (until it's too late), and is easy to caricature as science fiction.2) During the Cold War companies weren't selling nukes for profit. An anti-nuclear treaty constrains a weapon. An anti-AI treaty constrains a technology that can be used for coding, drug discovery, material science, physics and math, robotics, etc. Dual-use makes enforcement of a pause economically painful, and it will only get worse as diffusion of AI progresses.3) There is no incentive to keep making nukes more and more powerful beyond a certain point. If you can destroy your enemy's cities and military bases, it's good enough. But when it comes to making AI more and more intelligent, there is no "ok, intelligent enough" ceiling.Note that I am implicitly assuming that ASI will be far more intelligent than the greatest humans (no ceiling at the human level), which brings me to point 4.4) As of 2026, most people do not believe that a superintelligent AI will be created any time soon, if ever. Even among so-called "Godfathers of AI" there is no clear agreement. Bengio and Hinton have high p(doom), LeCun disagrees. Bengio and Hinton agree that ASI is (relatively) close, LeCun disagrees on that as well.5) A pause where defection is not detectable is meaningless. It is nearly impossible to conduct a nuclear explosion te