Modeling Concepts Probabilistically

As I come up to speed on John Wentworth and David Lorell’s work on natural abstraction, I’m filling in some of the gaps in their writing. Previously I posted about a Test Suite for Concepts. Today I’m going to talk about why they use a probabilistic frame when reasoning about concept representation in the minds of agents and the kind of funny way they throw around the word “latent."Working With Concepts So far, we have this very high-level idea of a “concept” in a “mind.” We want to start to think about this in a mathematical way, so we can reason about it more precisely.What kind of framework should we use?There’s considerable prior art in the area of concept representation from such disparate fields as cognitive science, psychology, neuroscience, good-old-fashioned AI (GOFAI), and machine learning. There’s quite a good Related Work section in the Natural Abstractions distillation article by Chan, Lang, and Jenner, so I won’t reprise all of that here.These various frameworks are solving different problems. Some of them are descriptive and grounded in, e.g., the biological details of exactly how information is retained in a brain made out of neurons, or the numerical details of how information is represented in a neural network. Other frameworks are quite high level and philosophical. So we’re not really comparing apples to apples when we look at all of these things next to each other.John’s main lens for thinking about concepts in minds is probabilistic and Bayesian, and it occupies a middle ground between the descriptive, low-level, hardware-specific frameworks and the abstract, philosophical, high-level frameworks. He’s interested in useful ways of modeling what’s going on inside the mind of an agent.Do agents “really” do Bayesian reasoning natively?Is John saying that all the various kinds of minds/agents literally implement probabilistic world models and update them in a purely Bayesian way? Is he saying that they’re all optimal utility maximizers?No, definitely