Somewhat Contra Ted Chiang on AI Consciousness

Ted Chiang recently published a piece in The Atlantic titled "No, Artificial Intelligence is Not Conscious." As a big fan of Chiang's fiction and someone with a deep interest in AI, I wanted to read it immediately. It's relatively short, and although I quote it extensively I do recommend that you also read it to provide context for the rest of this post.Chiang makes three major claims, and while I agree with one of them, I have different perspectives on the other two.I summarize his three claims as follows:LLMs are not consciousConsciousness requires a physical embodiment.Anthropic acts in many ways that suggest that it does not believe Claude is conscious, including in Claude's constitution.I mostly agree with (1), although I don't agree with Chiang's arguments for why this is the case. I definitely don't agree with (2) and I think that (3) has a benign and boring explanation.1. Chiang's Claim: LLMs Are Not ConsciousA Summary of Chiang's ArgumentChiang argues that LLMs are not conscious through an intuition pump. Specifically, he asks us to consider a LLM prompt that begins "The following is a conversation between Julius Caesar and Genghis Khan." He observes that we wouldn't suggest that the LLM has created conscious instantiations of Caesar and Khan here, even though this text is generated by the same set of weights and operations that LLMs use for other text generation. Now he moves to the next step, changing the prompt to "The following is a conversation between a helpful chatbot and a user," with the LLM generating the conversation for both the chatbot and the user. That is, the LLM operates in purely text completion mode and not interactively. He argues that nothing here has fundamentally changed just because we have replaced "Caesar and Khan" with "helpful chatbot and a user." Finally, he switches to the familiar chatbot model, where the LLM stops generating tokens for the user and instead the human enters text. Chiang claims that this is essentially the same