Algorithmic Perfection

This question has been wandering my mind a lot recently: What if someone decided to make a "model" that is optimized purely to take up infrastructure and create adversarial competitive pressure in the compute landscape? As the world becomes more and more efficient, I think there are three forces that are going to be way more consequential than what most people currently assume:Open-weights models that are easily fine-tunable on arbitrary targets like cyber offense and self-replication already exist.Current internet infrastructure is very fragmented, has tons of weak targets and has tons of resources that are politely "restricted" but not really hardened.Cybersecurity is an asymmetric task. Attackers have a significant advantage over defenders.Sure, you could argue an individual model without much compute capacity or significant can't really do much harm against hardened internet infrastructure, but I am not very sure whether someone could stop a 7B-70B parameter model finetuned for this task spreading over the internet somewhat like a more modern ILOVEYOU. I am not a pessimistic person. But the same dynamic that is making companies that would rather slow down the development of very capable AI systems have to keep competing and releasing models is the dynamic that could make hostile competition very convenient.I do not think cooperation and guardrails can fully mitigate this risk. Sure, you might argue, this is not yet a problem. "Clearly it hasn't happened". "Clearly, if some bad-spirited person had wanted to do this they would have already done it". This is not a very good counterargument. Think about something like antibiotic resistance. Soil bacteria had amoxicillin resistant genes very long before modern clinical antibiotics had been invented. The enzymes required to degrade these compounds was there, but since there was no competitive or evolutionary pressure to express these genes, the capacity was mostly inconsequential. The moment humans decided to carpet b