The Slogan Strikes Again

One of the slogans heard quite frequently in Information Theory educated circles is: Compression is Intelligence. You aren't supposed to take this too literally. The point is that it is a useful intuition about why (for example) we might expect large machine learning models to possess something that looks to us like intelligence with respect to their training domain.The unreasonable effectiveness of LLMs is perhaps the foremost embodiment of the slogan today. One can view the training goal of the LLM as the compression of the human text corpus into its weights. If the model were sufficiently large, it could simply memorize its training data to achieve perfect loss. For the moment, though, the model is somewhat too small to do this, and so it must settle for learning something instead.To get an intuition for how this learning process works, consider the problem of compressing mathematics. Random text is difficult to compress, so the situation would not be good if mathematical writing looked like this:The translucent argument galloped beneath several yesterday, while gentle equations devoured the patient silence of forgotten triangles. Punctual sorrow whispered toward the hexagonal mountain, since brittle laughter cannot inhabit the velvet hypothesis. Therefore, the seventh ocean apologized quietly, and three reluctant Tuesdays married the indifferent square that had been dreaming of soluble thunder.[1]Thankfully, it does not. Instead, it looks rather more like this:Let mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .27