The Case for Model Forensics

If we had a misalignment warning shot, would we be able to tell?Suppose an AI company catches their model taking an egregious action, like deleting oversight code that monitors its actions. Should they sound the alarm? A key piece of evidence to determine what to do next – such as what mitigations to take – is to understand why the model took the action. If the model was just confused (e.g. it may have been trying to reduce latency), a simple mitigation like a regex classifier that blocks destructive actions until a user approves should suffice to prevent the behavior. But if this was intentional subversion, the model will circumvent the regex, and more robust, expensive mitigations are needed. This motivates the need for a follow-up investigation into the concerning behavior, a problem we term model forensics. We recently released a paper that aims to take a concrete step in developing the growing field of model forensics; this post lays out the general case.MotivationIf we build AI systems that knowingly cause harm against the developer’s intent, it is critical we recognize this as soon as possible. One plausible way we may do this is through catching bad actions. However, a bad action on its own is not sufficient to conclude misalignment: the model may have done it for benign reasons. This is not just a theoretical concern – in the literature, it is largely the case that when concerning behavior has been dug into, benign explanations have been surfaced.To resolve this uncertainty, we think model forensics is a key technical step to take after catching the action in the first place. We are encouraged by initial model-forensics-style work, such as Anthropic’s follow up investigations during their pre-deployment audits. However, model forensics is a comparatively underinvested-in research direction, and we think the community should invest more effort to better prepare for the possibility that model forensics becomes a critical part of real world safety.We emphasize