Thoughts on 'Learning Mechanics'

Last month, I stumbled upon an optimistic-sounding paper: "There Will Be a Scientific Theory of Deep Learning". In this paper, we make the case that a scientific theory of deep learning is emerging. By this we mean a theory which characterizes important properties and statistics of the training process, hidden representations, final weights, and performance of neural networks. Historically, machine learning has been a field where theoretical understanding has lagged far behind empirical success. This paper argues that might soon start to change. The authors propose that we are gradually seeing the emergence of what they call learning mechanics---a mathematical understanding of the process by which machine learning models learn. And the newly-developing theory looks an awful lot like physics! We argue that the emerging theory is best thought of as a mechanics of the learning process, and suggest the name learning mechanics. We assert that learning mechanics should be a mathematical theory, grounded in first-principles calculations that closely predict empirics, reliant on well-tested approximations and assumptions, aiming for broad impact across the machine learning stack once it reaches maturity. The name "learning mechanics" was deliberately chosen to echo the various mechanics of physics. Examples of mechanics are: Classical mechanics: the study of how objects move and interact under the influence of forces. Statistical mechanics: the study of how the collective behavior of many microscopic particles gives rise to macroscopic phenomena like temperature and pressure. Quantum mechanics: the study of how matter and energy behave at the smallest scales, where physical quantities are quantized and governed by probabilistic laws. Taken together, these bodies of work share certain broad traits: they are concerned with the dynamics of the training process; they primarily seek to describe coarse aggregate statistics; and they emphasize falsifiable quantitative predictions.