Towards a Formal Scientific Epistemology

In my post “Why I’m not a Bayesian”, I argued that the Bayesian approach of assigning credences to propositions with binary truth values only works in simple and restricted domains. Instead, I claimed, a better approach to epistemology is to assign degrees of truth to models of the world.This approach is broadly inspired by science, which is the domain from which we have the most evidence about which epistemological practices allow us to solve very hard problems. We don’t currently have a complete theory of scientific epistemology, but we can identify some important differences between scientific epistemology and Bayesian epistemology. Central examples of Bayesian epistemology (such as Solomonoff induction) assume that the truth lies within the class of hypotheses being considered. Conversely, in central examples of scientific research, the truth is definitely not already under consideration: the main problem is to come up with any hypothesis that explains existing data.Another way of putting this point: Bayesian epistemology is entirely about empirical updates, whereas science is mostly about the process of constructing new theories. In some cases, once you’ve constructed a theory, you can be confident that it’s close to the truth merely from how well it fits existing data. But scientific theories are only fully accepted after they make successful advance predictions. That’s another difference compared with Bayesian epistemology, which treats retrodictions as equivalent to predictions.In general I think scientific epistemology is far superior as a guide for thinking about difficult problems (like AI alignment) than Bayesian epistemology. However, scientific epistemology has mostly been described informally—e.g. by Popper, Kuhn, Feyerabend, etc. Popper did attempt to formally define a metric for degrees of truth, but it wasn’t very successful. I’d like to be able to describe scientific epistemology as formally as we can describe Bayesian epistemology (and ideally to