How do we make uncertainty usable?

Most AI researchers put at least 5% on advanced AI causing human extinction or extreme disempowerment. The most cited scientists of all time are Yoshua Bengio and Geoffrey Hinton and they both put extinction at 10% or more. There is a distinct lack of global response to this kind of message from relevant scientists.One reason people purport to not be impressed by these chances of extinction is that they are subjective probabilities, not verified frequencies or arising from a more formal process.My natural response to this concern is to argue about this, since I think it is very wrong and misguided.But a different direction would be to just try to get admissible numbers on these things that some of our more normal processes of responding to risk can respond to. It’s going to be hard to get straight up frequentist numbers due to extinction being not very repeatable. But in other areas of life, we have systems that must deal with subjective probabilities and non-repeating circumstances, and take them seriously and quantitatively. The fact that a specific company can’t go bankcrupt a large enough number of times to get a good outside view on it happening in a given month doesn’t mean investors just write off that possibility as speculation. It gets a quantitative part in the price of that company. And the chance of it happening isn’t an output of a formal method—it’s a bunch of people’s guesses, from looking at a detailed situation. Somehow the subjective guesses about this and a multitude of other sources of future value and disvalue get processed into an extremely specific formal price number that official processes of the world take seriously. If the price of a company goes down because people think it might go bankrupt, a bunch of people will make different decisions as a result. And in the end, very few uncertainties in the world can be turned into clear frequencies—even if you have a disease that a million other people get every year, you aren’t the perfectly aver