AI will make biological extinction risks worse before it makes them better
An argument goes: If we don't build aligned artificial superintelligence, we risk driving ourselves extinct for some other reason. We should rush to build ASI quickly, in spite of the risks—the longer we wait, the more vulnerable we are to extinction from a different cause. Other than ASI, the biggest extinction risk is synthetic biology. Some lab could (accidentally or on purpose) develop a highly transmissible, 100% fatal super-plague that wipes out humanity. An aligned ASI could stop that from happening by shutting down dangerous biological research, or by developing advanced countermeasures that stop the spread of deadly infections. So the argument goes: We need to build ASI to save us from non-AI extinction risks. However, that argument doesn't work. In the near term, AI will make biological risks worse, not better. AI will accelerate scientific research, which will bring us closer to the level of knowledge necessary to build extinction-level pathogens. And in the long term, the way ASI eliminates biological x-risk is by taking control of the world. Cross-posted from my website. In the near term, AI makes biorisk worse Some people imagine that AI models would accelerate defensive research while refusing to assist with developing bioweapons. This plan has two minor issues and one fatal one. The first minor issue: Current AI model refusals are not robust, and there are workarounds to get information out of them for people who want to. It's very hard for AI developers to patch all holes, but the jailbreakers only need to find one. The second minor issue: Even if the leading AI developer makes their model safe and un-jailbreakable, at least one of their competitors will probably fail at that task. The fatal issue: It's not just about what AI assistants can do for humans. It's that AI accelerates the rate of scientific progress. As state of knowledge improves for humanity in general, it becomes possible for humanity to develop existentially risky pathogens, even if