On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regime

2020/05/27 by Stephan Wojtowytsch, Wojtowytsch, Stephan · 6 citations
Computer Science · Mathematics · #35F20 #35Q68 #49Q22 #68T07 #Analysis of PDEs (math.AP) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Neural Networks and Applications #Stochastic Gradient Optimization Techniques #cs.LG #math.AP #msc:35F20 #msc:35Q68 #msc:49Q22 #msc:68T07 #stat.ML

paper · pdf · doi:10.48550/arxiv.2005.13530

arxiv created 2020/05/27 · openalex publication_date 2020/05/27 · arxiv updated 2020/05/28 · openalex created_date 2022/07/26 · openalex updated_date 2026/07/28

Abstract

We describe a necessary and sufficient condition for the convergence to minimum Bayes risk when training two-layer ReLU-networks by gradient descent in the mean field regime with omni-directional initial parameter distribution. This article extends recent results of Chizat and Bach to ReLU-activated networks and to the situation in which there are no parameters which exactly achieve MBR. The condition does not depend on the initalization of parameters and concerns only the weak convergence of the realization of the neural network, not its parameter distribution.

Citations

• On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport
• Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks
• Analysis of the Gradient Descent Algorithm for a Deep Neural Network Model with Skip-connections
• Mean-Field Langevin Dynamics and Energy Landscape of Neural Networks
• A mean-field limit for certain deep neural networks
• Kolmogorov Width Decay and Poor Approximators in Machine Learning: Shallow Neural Networks, Random Feature Models and Neural Tangent Kernels
• Can Shallow Neural Networks Beat the Curse of Dimensionality? A mean field training perspective

Cited by

• Elliptic Regularity Theory in Barron Spaces and Applications to the Deep Ritz Method
• Optimal and Diffusion Transports in Machine Learning
• Phase Diagram of Dropout for Two-Layer Neural Networks in the Mean-Field Regime
• Acceleration via silver step-size on Riemannian manifolds with applications to Wasserstein space
• Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initialization
• On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regime

Related

• Convergence Analysis of Two-layer Neural Networks with ReLU Activation
• Mean-Field Analysis of Two-Layer Neural Networks: Global Optimality with Linear Convergence Rates
• Training Two-Layer ReLU Networks with Gradient Descent is Inconsistent
• Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks
• On the Proof of Global Convergence of Gradient Descent for Deep ReLU\n Networks with Linear Widths