The Principles of Deep Learning Theory

2021/06/18 by Daniel A. Roberts, Sho Yaida, Boris Hanin · 11 voices · 13 citations
Computer Science · Physics and Astronomy · #Neural Networks and Applications #Anomaly Detection Techniques and Applications #Statistical Mechanics and Entropy

paper · pdf · doi:10.1017/9781009023405

Abstract

This textbook establishes a theoretical framework for understanding deep learning models of practical relevance. With an approach that borrows from theoretical physics, Roberts and Yaida provide clear and pedagogical explanations of how realistic deep neural networks actually work. To make results from the theoretical forefront accessible, the authors eschew the subject's traditional emphasis on intimidating formality without sacrificing accuracy. Straightforward and approachable, this volume balances detailed first-principle derivations of novel results with insight and intuition for theorists and practitioners alike. This self-contained textbook is ideal for students and researchers interested in artificial intelligence with minimal prerequisites of linear algebra, calculus, and informal probability theory, and it can easily fill a semester-long course on deep learning theory. For the first time, the exciting practical advances in modern artificial intelligence capabilities can be matched with a set of effective principles, providing a timeless blueprint for theoretical research in deep learning.

Citations

• ImageNet classification with deep convolutional neural networks

Cited by

• From lazy to rich to exclusive task representations in neural networks and neural codes
• Fermions and Supersymmetry in Neural Network Field Theories
• Compiling to linear neurons
• A Primer on Deep Learning for Causal Inference
• Random neural networks in the infinite width limit as Gaussian processes
• NTKMTL: Mitigating Task Imbalance in Multi-Task Learning from Neural Tangent Kernel Perspective
• Quantum machine learning and quantum-inspired methods applied to computational fluid dynamics: a short review
• Statistics of correlations in nonlinear recurrent neural networks
• Feature Identification via the Empirical NTK
• AiDE-Q: Synthetic Labeled Datasets Can Enhance Learning Models for Quantum Property Estimation
• Neural Tangent Kernel Maximum Mean Discrepancy
• The GINN framework: a stochastic QED correspondence for stability and chaos in deep neural networks
• Viability of perturbative expansion for quantum field theories on neurons
• Criticality analysis of nuclear binding energy neural networks

Discussions

• The Principles of Deep Learning Theory [hn, 221 points, 139 comments]
• arxiv.org/pdf/2106.101... これが面白いという話を聞いたので読む [bsky, 4 points, 0 comments]
• The Principles of Deep Learning Theory [hn, 4 points, 0 comments]
• For those who, like me, are involved with #DeepLearning, follow the book: "Principles of Deep Learning Theory" by Roberts and Yaida. Tks #KirkBorne arxiv.org/pdf/2106.101... #datascience #neuralnets # [bsky, 4 points, 1 comments]
• The Principles of Deep Learning Theory (2021) [hn, 2 points, 0 comments]
• The Principles of Deep Learning Theory [hn, 2 points, 0 comments]
• I'm sorry but I'm a simple beast. Anything that combines neural networks with effective field theory and gaussian integrals is going to get my attention. arxiv.org/abs/2106.10165 [bsky, 1 points, 0 comments]
• Not specific to transformers, but this might be closer to what you're looking for: arxiv.org/abs/2106.10165 [bsky, 1 points, 1 comments]
• The principles of Deep Learning theory book is also a test gold mine. Generally, I think NTK is a good probabilistic framework for designing evaluations of DL models. [bsky, 1 points, 1 comments]
• - study existing AI systems using methods from physics (the principles of deep learning theory book arxiv.org/abs/2106.10165 statistical physics arxiv.org/abs/2506.04374 ,...), (2/n) [bsky, 0 points, 1 comments]
• Before answering your question, I want to note that a reference the physicist finds interesting may be different from what the computer scientist / machine learning researcher finds interesting. Not b [stackexchange, 0 points]

Related

• Mathematical theory of deep learning
• The Foundations of Deep Learning with a Path Towards General Intelligence
• There Will Be a Scientific Theory of Deep Learning
• Principles and Practice of Deep Representation Learning: or a Mathematical Theory of Memory
• The many faces of deep learning