Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks

2019/02/07 by Phan-Minh Nguyen, Nguyen, Phan-Minh · 2 citations
Computer Science · Physics and Astronomy · #Disordered Systems and Neural Networks (cond-mat.dis-nn) #FOS: Computer and information sciences #FOS: Physical sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Model Reduction and Neural Networks #Neural Networks and Applications #Statistical Mechanics (cond-mat.stat-mech) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.1902.02880

openalex publication_date 2019/02/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Can multilayer neural networks -- typically constructed as highly complex structures with many nonlinearly activated neurons across layers -- behave in a non-trivial way that yet simplifies away a major part of their complexities? In this work, we uncover a phenomenon in which the behavior of these complex networks -- under suitable scalings and stochastic gradient descent dynamics -- becomes independent of the number of neurons as this number grows sufficiently large. We develop a formalism in which this many-neurons limiting behavior is captured by a set of equations, thereby exposing a previously unknown operating regime of these networks. While the current pursuit is mathematically non-rigorous, it is complemented with several experiments that validate the existence of this behavior.

Citations

• ImageNet Large Scale Visual Recognition Challenge
• Understanding deep convolutional networks
• No bad local minima: Data independent training error guarantees for\n multilayer neural networks
• Opening the Black Box of Deep Neural Networks via Information
• Recovery Guarantees for One-hidden-layer Neural Networks
• The loss landscape of overparameterized neural networks
• Deep Convolutional Networks as shallow Gaussian Processes
• A Convergence Theory for Deep Learning via Over-Parameterization
• Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel

Cited by

• A Note on the Global Convergence of Multilayer Neural Networks in the\n Mean Field Regime
• On the Empirical Neural Tangent Kernel of Standard Finite-Width Convolutional Neural Network Architectures

Related

• A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networks
• Limiting fluctuation and trajectorial stability of multilayer neural\n networks with mean field training
• Global Convergence of Three-layer Neural Networks in the Mean Field\n Regime
• A Note on the Global Convergence of Multilayer Neural Networks in the\n Mean Field Regime
• Mean Field Analysis of Deep Neural Networks