Exploring Known Unknowns in the AI Regulatory Landscape

The AI regulatory space is a rapidly developing and maturing one, and while a lot of work has recently been done to draft new bills and establish new frameworks, there’s still a ton we don’t know about the space. This post aims to quantify and qualify some of the “known unknowns” about the space, including what I believe to be one of the most consequential unknown variables: The lag time from when new AI capabilities are announced to when regulatory bodies acknowledge or pass legislation targeted at them.My research into this topic covered six different categories:1) Regulatory adequacy2) Regulatory causality3) Soft law effectiveness4) Incremental vs novel capabilities5) Preceding disasters6) Regulatory lag7) Miscellaneous Measurement GapsThe Definition of “Regulatory Adequacy”As far as I can tell, there’s no definition of “regulatory adequacy”. That is, there’s no agreed on definition for what “adequate” governance would actually look like.Granted, it’s quite hard to know what “adequate” looks like in governance as you’re generally trying to optimize for something not happening. But the lack of any real standard does have costs as it makes claims about “widening gaps” between capability development and governance rather fuzzy. No standardized metrics exist for determining if a framework or law actually improves the public trust, increases model safety, or reduces AI-related harms. And without legible standards for what counts as success it’s difficult to work backwards and determine what types of advocacy or legislation campaigns are actually effective at achieving their goals.Most databases or indices of AI governance such as OECD.AI or Stanford HAI measure just the quantity of governance, like the number of laws passed or how many strategies/frameworks have been published. For instance, the OECD.AI Index combines 28 indicators that track OECD’s AI principles, but these measure only policy implementation and capacity. They don’t track how effective policies are at