The Dual-Use Gap

TLDR: So there has been recent discourse on 𝕏, and recent news of major cyber attacks that were done with the help of AI. The missing frame here is the dual-use gap: as AI models become more capable, they create more upside for defenders and more downside for attackers. The gap between the benefits and the harmful effects is getting wider. I know that sounds obvious, but I think people are underestimating the second-order effects of this. The dual-use gap makes small failures feel less small. A compromised account, a bad package, or one missed vulnerability can suddenly have a much larger blast radius, because attackers can use AI to move faster, automate more, and chain together mistakes that would have been harder to exploit before. And the usual response is "good AI will defend against bad AI," but who is actually guaranteeing that the defensive AI finds every path before the attacking AI does? And also who is guaranteeing that it defends everyone?This is cross-posted on EA Forum; I’ll try to keep an eye on discussion in both places.So there has been recent discourse on 𝕏, and recent news of major cyber attacks that were done with the help of AI.I think a lot of the discourse is getting stuck on the wrong question. The debate is usually something like:One side says we need more safety precautions around open source and closed source model releases, because cyber capable models make vulnerability discovery and exploit development cheaper.The other side says this is fine because cyber is dual-use. As long as the "good" AI is better than the "bad" AI, defenders will win, like how Jensen Huang mentions in this podcast clip.I get why the second view is attractive. It's optimistic. The defender does have advantages sometimes. Mozilla's blog post on Mythos finding a ton of bugs is probably the most supporting of this view, and the conclusion was basically that maybe the defects are finite and we can finally find them all with AI. But my worry is that this view subtly