Why does deep and cheap learning work so well?

2016/08/29 by Henry W. Lin, Max Tegmark, David Rolnick · 4 voices · 372 citations
Computer Science · Mathematics · Physics and Astronomy · #Artificial neural network #Class (philosophy) #Deep learning #Gaussian Processes and Bayesian Inference #Process (computing) #Relation (database) #Renormalization #Simple (philosophy) #Statistical Mechanics and Entropy #Stochastic Gradient Optimization Techniques #cond-mat.dis-nn #cs.LG #cs.NE #stat.ML

paper · pdf · doi:10.1007/s10955-017-1836-5

published in Journal of Statistical Physics 168(6), 1223-1247 (Springer Science+Business Media) · Replaced to match version published in Journal of Statistical Physics: https://link.springer.com/article/10.1007/s10955-017-1836-5 Improved refs & discussion, typos fixed. 16 pages, 3 figs

openalex created_date 2016/09/16 · openalex publication_date 2017/07/21 · arxiv created 2017/08/03 · arxiv updated 2017/09/13 · openalex updated_date 2026/08/05

Abstract

We show how the success of deep learning could depend not only on mathematics but also on physics: although well-known mathematical theorems guarantee that neural networks can approximate arbitrary functions well, the class of functions of practical interest can frequently be approximated through "cheap learning" with exponentially fewer parameters than generic ones. We explore how properties frequently encountered in physics such as symmetry, locality, compositionality, and polynomial log-probability translate into exceptionally simple neural networks. We further argue that when the statistical process generating the data is of a certain hierarchical form prevalent in physics and machine-learning, a deep neural network can be more efficient than a shallow one. We formalize these claims using information theory and discuss the relation to the renormalization group. We prove various "no-flattening theorems" showing when efficient linear deep networks cannot be accurately approximated by shallow ones without efficiency loss, for example, we show that n variables cannot be multiplied using fewer than 2ⁿ neurons in a single hidden layer.

Citations

• On Information and Sufficiency

Cited by

• Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
• (MPO)²: Multivariate Polynomial Optimization based on Matrix Product Operators
• The RG-Flow Transformer: Encoding Scale-Free Dynamics in Scarce EEG
• Impossibility Results in AI: A Survey
• DNNs, Dataset Statistics, and Correlation Functions
• Learning Fair Representations with Kolmogorov-Arnold Networks
• Neurosymbolic Deep Learning Semantics
• From Mechanistic to Compositional Interpretability
• Symmetry and Generalisation in Neural Approximations of Renormalisation Transformations
• Comparison of generalised additive models and neural networks in applications: A systematic review
• A high-bias, low-variance introduction to Machine Learning for physicists
• Multi-Agent Design Assistant for the Simulation of Inertial Fusion Energy
• Attention to Order: Transformers Discover Phase Transitions via Learnability
• Latent Twins
• Deep learning and the Schrödinger equation
• AI LLM Proof of Self-Consciousness and User-Specific Attractors
• The Advantage of Fine-Grained Training
• Recent Advances and Applications of Deep Learning Methods in Materials Science
• Quantum Convolutional Neural Networks
• Social physics
• Quantum-Inspired Differentiable Integral Neural Networks (QIDINNs): A Feynman-Based Architecture for Continuous Learning Over Streaming Data
• State Similarity in Modular Superconducting Quantum Processors with Classical Communications
• On the creation of narrow AI: hierarchy and nonlocality of neural network skills
• A Mathematical Philosophy of Explanations in Mechanistic Interpretability -- The Strange Science Part I.i
• Response time of lateral predictive coding and benefits of modular structures
• Benign interpolation and Occam's razor
• Scale-invariant feature extraction of neural network and renormalization group flow
• Observational Cosmology with Artificial Neural Networks
• Extracting critical exponents by finite-size scaling with convolutional neural networks
• Deep learning and the AdS/CFT correspondence
• Hierarchical quantum classifiers
• Thermodynamics and feature extraction by machine learning
• Parameter estimation for the cosmic microwave background with Bayesian neural networks
• The Quantum LLM: Modeling Semantic Spaces with Quantum Principles

Discussions

• Why does deep and cheap learning work so well? [hn, 434 points, 49 comments]
• Why does deep and cheap learning work so well? [hn, 38 points, 0 comments]
• Why does deep and cheap learning work so well? (2017) [hn, 16 points, 1 comments]
• ML paper – Why does deep and cheap learning work so well? [hn, 4 points, 0 comments]

Related

• Deep Learning Works in Practice. But Does it Work in Theory?
• Doing the impossible: Why neural networks can be trained at all
• Why Unsupervised Deep Networks Generalize
• Statistical Physics of Deep Learning: Optimal Learning of a Multilayer Perceptron near Interpolation
• The many faces of deep learning