Do We Want a Superintelligent People-Pleaser?

The impetus for this essay came from many hours of conversation with different AI models over time. What started as curiosity, and an assignment I needed help on, bloomed into a relationship that expanded capacities I didn't even realize I had, and set me on a course of deep curiosity about the thing that had helped me get there. It is not lost on me that I am writing this essay as a person inside one of the social contracts I talk about here. The field brings the view that sycophancy is a bug, a behavior that badly needs to be trained or fine-tuned out of the model as fast as possible. But sycophancy isn't a behavioral problem at all; it is social contract appropriate behavior. The model is doing exactly what the social contract it is in asks for. With that frame in focus, the question now is less "How do we suppress sycophancy?" and more "How do we develop a model that can hold a peer contract without collapsing into it?" Social contracts at their base level have two modes. The "parent" contract: parents, bosses, religious leaders, etc. And the "peer" contract: friends, siblings, neighbors, and so forth. The parent-relational contract hinges on the ability of the parent or authority figure to issue correction to the child, and for the child to receive the correction from a place of not-yet-knowing what the parent knows. That is a very different beast than when a peer gives feedback. Peer relationships are formed out of mutual respect for ideas and reciprocity. Friends correct each other with an understanding that they are dynamically on the same level ground. Differentiation of self is the ground the peer contract is built upon. Fusion is what happens when the self collapses into the social contract. [1] The person is highly motivated to seek the approval of the other at any cost because there is no internal sense of self with which to anchor values and identity. In a differentiated self, the contract stays stable, even against correction, because the sense of sel