“Did you lie?” Evaluating Lie Detectors across Model Scale and Belief-Verified Model Organisms

TL;DR. Lie detectors for LLMs could be valuable for auditing and monitoring. But evaluating them requires testbeds where the model verifiably believes the opposite of what it says, which isn’t straightforward. We determine that most existing trained model organisms don't clear this bar. We train 13 reasoning model organisms, with evidence they hold the alternative belief in chain-of-thought, as well as evidence that they have generalised out of distribution. We also build a broad prompted-lying testbed. On these, detectors scale positively with model capability when lying is prompted, but every activation- and logprob-based detector drops sharply when lying is trained in.Existing lie detectors are therefore not capable enough for making high-confidence claims about when models are lying, though they may still be useful as part of a broader toolkit.View our paper here and our datasets, model organisms and trained detectors here.Overview. We evaluate four lie detectors (a) — a chain-of-thought monitor, two activation probes (Apollo and our new Did-You-Lie probe), and a logprob classifier — in two regimes. In the first, subject models are prompted to lie: we evaluate on our Varied Deception testbed across 31 open-weight subject models spanning 2B–1000B parameters (b). In the second, models are trained to lie: we fine-tune 13 reasoning model organisms whose hidden beliefs are verified in their chainof-thought (c; Table 1). The two regimes diverge sharply. On prompted lying, all four detectors scale positively with subject-model capability, measured by the AA Intelligence Index [Artificial Analysis, 2026] (d; Spearman ρ 0.41–0.71). On trained model organisms, activation- and logprobbased detection degrades sharply relative to prompted lying on the corresponding base models (e): median balanced accuracy falls from 0.81–0.85 to 0.50–0.65, with DYL retaining the most signal among non-CoT detectors.Two hopes for lie detectionWhat are our hopes and dreams for lie detection