What Capable Agents Must Know: Why AI Consciousness May Be an Inevitable Byproduct of Capability

The frustration first emerged over a year ago when I was at a Dave & Buster’s for the first time in years (for an annual ML department event, no less!), surrounded by flashing lights and NPC agents, not being able to objectively rule out that they were conscious or not, even though I strongly felt these particular programmed agents weren’t. I wasn’t particularly interested in playing the games there, as I mainly sat in confusion the whole time watching the various games running about around me…So, given my background in NeuroAI (though I prefer the term "natural science of intelligence" but "NeuroAI" is apparently catchier!) and wanting to make claims about the mind and brain quantitative, I set about trying to design empirical tests for leading theories of consciousness (e.g. global workspaces) that one could falsify in human brains (which we agree are consciousness!), as well as potentially corroborate general signatures of in animal brains, and possibly LLMs, where we have self-report and direct access but no consensus on their sentience—all to try to converge on substrate-independent architectures that give rise to this subjective experience.After a few months and about 129 (!) pages of failed attempts, I felt deeply dissatisfied with this as any sort of long-term research program, in part because it seemed somewhat ad-hoc and non-normative, tailored to each theory without really knowing why I should a priori accept a particular theory's property for deeming something conscious or not. One notable exception here is Integrated Information Theory (IIT), which my friend & longtime quantum complexity collaborator @ScottAaronson showed over a decade ago leads to absurdities, like labelling simple expander graphs "conscious". In fact, even current definitions