The Modern Mathematics of Deep Learning

2021/05/09 by Julius Berner, Philipp Grohs, Gitta Kutyniok +1 · 6 voices · 125 citations
Computer Science · Engineering · Mathematics · Physics and Astronomy · Psychology · #Artificial intelligence #Cognitive science #Computer science #Convexity #Curse of dimensionality #Data science #Deep learning #Engineering #Field (mathematics) #Generalization #Mathematics #Mathematics education #Model Reduction and Neural Networks #Neural Networks and Applications #Psychology #Pure mathematics #Sparse and Compressive Sensing Techniques #Task (project management)

paper · pdf · doi:10.1017/9781009025096.002

published in Cambridge University Press eBooks, 1-111 (Cambridge University Press)

openalex publication_date 2022/11/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

We describe the new field of the mathematical analysis of deep learning. This field emerged around a list of research questions that were not answered within the classical framework of learning theory. These questions concern: the outstanding generalization power of overparametrized neural networks, the role of depth in deep architectures, the apparent absence of the curse of dimensionality, the surprisingly successful optimization performance despite the non-convexity of the problem, understanding what features are learned, why deep architectures perform exceptionally well in physical problems, and which fine aspects of an architecture affect the behavior of a learning task in which way. We present an overview of modern approaches that yield partial answers to these questions. For selected approaches, we describe the main ideas in more detail.

Citations

• A Stochastic Approximation Method
• Stochastic Estimation of the Maximum of a Regression Function
• Analytic extensions of differentiable functions defined in closed sets

Cited by

• Stochastic restarting with multiple restart conditions
• A machine learning approach to predict L-edge x-ray absorption spectra of light transition metal ion compounds
• Discretization-independent multifidelity operator learning for partial differential equations
• PINN-DG: Residual neural network methods trained with Finite Elements
• Categorical and geometric methods in statistical, manifold, and machine learning
• FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference
• Solomonoff induction
• Benign interpolation and Occam's razor

Discussions

• The Modern Mathematics of Deep Learning [hn, 276 points, 70 comments]
• The Modern Mathematics of Deep Learning [hn, 23 points, 2 comments]
• Wer mathematisch etwas vorbelastet ist und sich nur für die Mathematik hinter Deep Learning interessiert, ohne gleich ein ganzes Buch durcharbeiten zu müssen, findet hier eine kompakte und systematisc [bsky, 12 points, 0 comments]
• The Modern Mathematics of Deep Learning [hn, 3 points, 0 comments]
• The modern mathematics of deep learning [hn, 2 points, 0 comments]
• Can you unpack this a bit? Some argue that large models work well in machine learning because of the mysterious fact that gradient descent improves at scale, despite non-convexity (arxiv.org/pdf/2105. [bsky, 1 points, 1 comments]

Related

• Mathematics of Deep Learning
• Mathematical theory of deep learning
• Deep Learning: An Introduction for Applied Mathematicians
• Principles and Practice of Deep Representation Learning: or a Mathematical Theory of Memory
• Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges