Hyperstition as the Natural Enemy of Rationality

If the box contains a diamond,I desire to believe that the box contains a diamond;If the box does not contain a diamond right now, but will contain a diamond if I believe there is a diamond,Uh...Holding unfounded beliefs might sometimes, by some cruel irony, produce better outcomes than being rational.(This post was inspired by a couple cases where the causal effect of belief seems hand-waved away in the Sequences.)"Diseased Thinking"In this essay, Scott suggests that a consequentialist model deals with the question of whether to moralize issues like obesity better than a definitional argument over whether it is a "disease" or not. If it benefits the person, you moralize; otherwise you let them resort to medical interventions guilt-free.But there's this annoying feature of morality where most people feel like it has to be absolute to be worth acting on.[1] You can't just say "we should only guilt people if it would benefit them". The person is either guilty or not guilty; you can't pragmatically decide whether they're guilty or not. The consequentialist frame debuffs the power of moral pressure.Some individuals, who would have gotten their act together if everyone bought into the old-fashioned guilt and willpower model, will now take a medical way out, making them subject to side effects from the medication or procedure. On net, this could outweigh the benefit of lifting guilt from those for whom willpower is not the deciding factor. The consequentialist framework might actually produce worse equilibria than the traditional one. What I'm getting at is that optimal solutions might involve people believing something unfounded, and rationality will never converge to this solution. This creates an inherent tension where conservatives reach better equilibria because they can believe in things like God or that a marriage is a divinely sanctioned mutual partnership.Imagine a society where every member believes they will be punished eternally if they intentionally harm anyo