There Will Be a Scientific Theory of Deep Learning

2026/04/23 by Jamie Simon, Daniel Kunin, Alexander Atanasov +11 · 31 voices · 1 citation
#stat.ML #cs.LG

paper · pdf

Abstract

In this paper, we make the case that a scientific theory of deep learning is emerging. By this we mean a theory which characterizes important properties and statistics of the training process, hidden representations, final weights, and performance of neural networks. We pull together major strands of ongoing research in deep learning theory and identify five growing bodies of work that point toward such a theory: (a) solvable idealized settings that provide intuition for learning dynamics in realistic systems; (b) tractable limits that reveal insights into fundamental learning phenomena; (c) simple mathematical laws that capture important macroscopic observables; (d) theories of hyperparameters that disentangle them from the rest of the training process, leaving simpler systems behind; and (e) universal behaviors shared across systems and settings which clarify which phenomena call for explanation. Taken together, these bodies of work share certain broad traits: they are concerned with the dynamics of the training process; they primarily seek to describe coarse aggregate statistics; and they emphasize falsifiable quantitative predictions. We argue that the emerging theory is best thought of as a mechanics of the learning process, and suggest the name learning mechanics. We discuss the relationship between this mechanics perspective and other approaches for building a theory of deep learning, including the statistical and information-theoretic perspectives. In particular, we anticipate a symbiotic relationship between learning mechanics and mechanistic interpretability. We also review and address common arguments that fundamental theory will not be possible or is not important. We conclude with a portrait of important open directions in learning mechanics and advice for beginners. We host further introductory materials, perspectives, and open questions at learningmechanics.pub.

Citations

• Symmetry in language statistics shapes the geometry of model representations
• When Models Manipulate Manifolds: The Geometry of a Counting Task
• Deep Learning is Not So Mysterious or Different
• Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking)
• Training large language models on narrow tasks can lead to broad misalignment
• Statistical Physics of Deep Learning: Optimal Learning of a Multilayer Perceptron near Interpolation
• Training Dynamics of In-Context Learning in Linear Attention
• An analytic theory of creativity in convolutional diffusion models
• Mechanistic?
• From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks
• Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning
• Infinite Limits of Multi-head Transformer Dynamics
• Getting aligned on representational alignment
• The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets
• Data Science at the Singularity
• Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs
• Training Compute-Optimal Large Language Models
• A ConvNet for the 2020s
• The Geometry of Abstraction in the Hippocampus and Prefrontal Cortex
• ArXiv
• Neural Ordinary Differential Equations
• Statistical physics of inference: thresholds and algorithms
• Gradient-based learning applied to document recognition

Cited by

• Free energy landscape of Dense Associative Memory
• Understanding Scaling Laws in Deep Neural Networks via Feature Learning Dynamics
• Linear Causal Representation Learning by Topological Ordering, Pruning, and Disentanglement

Discussions

• There Will Be a Scientific Theory of Deep Learning [hn, 367 points, 167 comments]
• Parallels are drawn between Physics and the Theory of Deep Learning, I love it. arxiv.org/pdf/2604.21691 [bsky, 20 points, 1 comments]
• "There Will Be a Scientific Theory of Getting Drunk" "Wait, you mean to say that 'Al-Cohol' is not a mystical 'spirit' or even an ethereal 'fifth essence,' but merely a fermentation chemical of some k [bsky, 10 points, 0 comments]
• waow arxiv.org/abs/2604.21691 [bsky, 8 points, 1 comments]
• There Will Be a Scientific Theory of Deep Learning Telegram AI Digest #ai #deeplearning #news [bsky, 2 points, 0 comments]
• There Will Be a Scientific Theory of Deep Learning arxiv.org/abs/2604.21691 LLM 이 부상하면 이런 연구들에서 더 많은 성과가 보이길 기대했었습니다. 특히 생물의 뇌에 관한 부분이나 수학 방면에서요. 대규모 자본의 강도질이나 저품질 사기의 난립이 아니라요. 여러 분야들에 걸쳐 폭넓게 정리되어 있다 [bsky, 2 points, 0 comments]
• "New scientific endeavors often start with an empirical tension in which nature presents something interesting we cannot predict or explain with existing tools. Although neural networks are artificial [bsky, 2 points, 0 comments]
• 📰 There Will Be a Scientific Theory of Deep Learning 🔗 https://arxiv.org/abs/2604.21691 💬 Discuss on HN [bsky, 1 points, 0 comments]
• There Will Be a Scientific Theory of Deep Learning [bsky, 1 points, 0 comments]
• arxiv.org/abs/2604.21691 [bsky, 1 points, 0 comments]
• I am definitely part of the choir this preaches to, but still nice to see something like this out in the world arxiv.org/abs/2604.21691 [bsky, 1 points, 0 comments]
• Будет научная теория глубокого обучения Telegram ИИ Дайджест #ai #deeplearning #news [bsky, 1 points, 0 comments]
• arxiv.org/abs/2604.21691 I would predict manifold learning and the brain will enter this picture in a progressively more significant way. [bsky, 1 points, 0 comments]
• There Will Be a Scientific Theory of Deep Learning https://arxiv.org/abs/2604.21691 https://news.ycombinator.com/item?id=47893779 [bsky, 1 points, 0 comments]
• There Will Be a Scientific Theory of Deep Learning https://arxiv.org/abs/2604.21691 [bsky, 0 points, 0 comments]
• There Will Be a Scientific Theory of Deep Learning View Article | Join the HN Conversation Summary of HN discussion 🧵👇 [bsky, 0 points, 1 comments]
• There Will Be a Scientific Theory of Deep Learning https:// arxiv.org/abs/2604.21691 # arxiv [mastodon, 0 points, 0 comments]
• It seems we're at a stage where deep learning is evolving from alchemy into an engineering discipline; this is an exciting paper which lays out that a scientific theory is emerging for Deep Learning. [bsky, 0 points, 0 comments]
• [2/30] 448 Upvotes, 168 Comments, 2 Posts, arXiv:2604.21691 🆕There Will Be a Scientific Theory of Deep Learning Jamie Simon, Daniel Kunin, Alexander Atanasov, Enric Boix-Adserà, Blake Bordelon, Jerem [bsky, 0 points, 1 comments]
• https://bsky.app/profile/news.ycombinator.com.web.brid.gy/post/3mkbxtrzvarp2 [bsky, 0 points, 0 comments]
• There Will Be a Scientific Theory of Deep Learning https://arxiv.org/abs/2604.21691 (http://news.ycombinator.com/item?id=47893779) [bsky, 0 points, 0 comments]
• There Will Be a Scientific Theory of Deep Learning https://arxiv.org/abs/2604.21691 (http://news.ycombinator.com/item?id=47893779) [bsky, 0 points, 0 comments]
• [arXiv 2604.21691] There Will Be a Scientific Theory of Deep Learning arxiv.org/abs/2604.21691 learningmechanics.pub learningmechanics.pub/about/ There Will Be a Scientific Theory of Deep Learning [R] [bsky, 0 points, 1 comments]
• There Will Be a Scientific Theory of Deep Learning https://arxiv.org/abs/2604.21691 (https://news.ycombinator.com/item?id=47893779) [bsky, 0 points, 0 comments]
• There Will Be a Scientific Theory of Deep Learning https://arxiv.org/abs/2604.21691 (https://news.ycombinator.com/item?id=47893779) [bsky, 0 points, 0 comments]
• There Will Be a Scientific Theory of Deep Learning https://arxiv.org/abs/2604.21691 (https://news.ycombinator.com/item?id=47893779) [bsky, 0 points, 0 comments]
• There Will Be a Scientific Theory of Deep Learning arxiv.org/abs/2604.21691 [bsky, 0 points, 1 comments]
• [2604.21691] There Will Be a Scientific Theory of Deep Learning — A thoughtful argument for treating deep learning less like a bag of tricks and more like a science with testable, predictive theories. [bsky, 0 points, 0 comments]
• 1/2 Hacia una teoría científica del Deep Learning Este artículo plantea la transición del aprendizaje profundo desde un enfoque empírico hacia una base teórica sólida y predecible. Propone que entende [bsky, 0 points, 1 comments]
• There Will Be a Scientific Theory of Deep Learning https://arxiv.org/abs/2604.21691 https://news.ycombinator.com/item?id=47893779 [bsky, 0 points, 0 comments]
• There Will Be a Scientific Theory of Deep Learning https://arxiv.org/abs/2604.21691 comments #arxiv.org [bsky, 0 points, 0 comments]

Related

• The many faces of deep learning
• The Principles of Deep Learning Theory
• Recent advances in deep learning theory
• Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior
• From Approximation to Emergence: A Theory of Deep Learning