What can you do with barely any data?

(Epistemic Status: Exploring a cool technique I came across. Usefulness is debatable, but plausibly decent if you need something you can do in your head. Some of the math is intentionally handwavey, but I think the basic points are robust.)A little while ago, I was reading How to Measure Anything by Douglas Hubbard. For the most part, I found that the book, though interesting, mainly covered ideas and concepts I already knew, such as ‘value of information’, A/B testing, linear regression, and Bayesian forecasting. However, there was one fascinating exception that, after learning, I felt I had to share and investigate, thanks to its remarkable combination of incredible simplicity and strong mathematical guarantees.The MethodThe method, as I said, is remarkably simple. To understand it, we can first ask the question: What is the probability that n random, independently chosen samples are all above the population median? The answer to this question is pretty straightforward. Since there is a one-half chance that a single sample is above the median (by the definition of the median[1]), the probability that all of the independent samples are above the median is ½ raised to the power of n: (½)n.Taking this result, we can then say, by symmetry, that the probability that n random, independently chosen samples are all below the population median is also ½ raised to the power of n: (½)n.Now, since both of these possibilities are mutually exclusive, we can say that the probability that either all the n samples are above the median or all the n samples are below the median is just the sum of the two individual probabilities, which is just ½ raised to the power of n minus one: (½)n-1.This result might not seem particularly interesting, but if we switch to describing the complement (i.e., the event that neither of the conditions above is true), then this takes on a very different meaning. Specifically, it says that for n random, independently chosen samples, the probability that