Scoopfeeds — Intelligent news, curated.
agentic-ai

Many individual CEVs are probably quite bad

LessWrong · May 6, 2026, 8:18 PM

I was thinking about Habryka's article on Putin's CEV, but I am posting my response here, because the original article is already 3 weeks old.I am not sure how exactly a person's CEV is defined. "If we knew everything and could self-modify" seems potentially sensitive to the precise chronological order of "realizing things" and "self-modification".Like, imagine Hitler getting the godlike powers of knowledge and self-control. If he gets the perfect knowledge of economy, sociology and psychology first, he could go like: "Oh, now I realize that the things I blamed on the Jews are actually caused by something else. How embarrassing. No more anti-semitism, but I better erase everyone's memory first."But it is also possible that he gets the self-control first, and he realizes that there is such a thing as value drift, and thinks: "Oh my, this could accidentally make me more similar to the Jews. I better hardcode the Nazi ideals in myself immediately, and also give myself blond hair and blue eyes." And using the superior knowledge, he hardcodes the Nazi values in himself so that they are reflectively stable and survive all updates.So, Hitler's CEV seems to depend on the technical details, in which order he gets the new knowledge and the new skills. He could end up either CEV-nice or a CEV-monster.Seems like "knowledge first, self-modification next" is the preferred order, but that kinda assumes perfect rationality at the beginning. I mean, perfect knowledge without perfect rationality would probably be prone to confirmation bias and other biases. So we might want perfect rationality (or merely improved rationality) first, but making ourselves more rational is already in the realm of self-modification.Second, it seems to me that Habryka chooses between two models in the article: Either everyone is CEV-nice, or almost everyone is CEV-nice but a few people such as Putin are rare CEV-monsters. Then he concludes, in my opinion correctly given the premises, that Putin doesn't se

Article preview — originally published by LessWrong. Full story at the source.
Read full story on LessWrong → More top stories
Aggregated and edited by the Scoop newsroom. We surface news from LessWrong alongside other reporting so you can compare coverage in one place. Editorial policy · Corrections · About Scoop