Particle Dual Averaging: Optimization of Mean Field Neural Networks with Global Convergence Rate Analysis

2020/12/31 by Atsushi Nitanda, Nitanda, Atsushi, Denny Wu +3 · 1 citation
Computer Science · Engineering · Physics and Astronomy · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Model Reduction and Neural Networks #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2012.15477

openalex publication_date 2020/12/31 · openalex created_date 2021/01/05 · openalex updated_date 2026/07/28

Abstract

We propose the particle dual averaging (PDA) method, which generalizes the dual averaging method in convex optimization to the optimization over probability distributions with quantitative runtime guarantee. The algorithm consists of an inner loop and outer loop: the inner loop utilizes the Langevin algorithm to approximately solve for a stationary distribution, which is then optimized in the outer loop. The method can thus be interpreted as an extension of the Langevin algorithm to naturally handle nonlinear functional on the probability space. An important application of the proposed method is the optimization of neural network in the mean field regime, which is theoretically attractive due to the presence of nonlinear feature learning, but quantitative convergence rate can be challenging to obtain. By adapting finite-dimensional convex optimization theory into the space of measures, we analyze PDA in regularized empirical / expected risk minimization, and establish quantitative global convergence in learning two-layer mean field neural networks under more general settings. Our theoretical results are supported by numerical simulations on neural networks with reasonable size.

Citations

• Generalization of an Inequality by Talagrand and Links with the Logarithmic Sobolev Inequality
• Smooth minimization of non-smooth functions
• Primal-dual subgradient methods for convex problems
• Non-convex learning via Stochastic Gradient Langevin Dynamics: a\n nonasymptotic analysis
• A Convergence Theory for Deep Learning via Over-Parameterization
• Mean-Field Langevin Dynamics and Energy Landscape of Neural Networks
• Non-asymptotic convergence analysis for the Unadjusted Langevin Algorithm
• Sampling as optimization in the space of measures: The Langevin dynamics as a composite optimization problem
• Global Convergence of Langevin Dynamics Based Algorithms for Nonconvex Optimization
• Rapid Convergence of the Unadjusted Langevin Algorithm: Isoperimetry\n Suffices
• Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networks
• Gradient Descent can Learn Less Over-parameterized Two-layer Neural Networks on Classification Problems
• On the Convergence of Langevin Monte Carlo: The Interplay between Tail Growth and Smoothness
• Global Convergence of Three-layer Neural Networks in the Mean Field\n Regime

Cited by

• Mean-Field Generalisation Bounds for Learning Controls in Stochastic Environments

Related

• Mean-Field Analysis of Two-Layer Neural Networks: Global Optimality with Linear Convergence Rates
• Improved Particle Approximation Error for Mean Field Neural Networks
• Mean-Field Langevin Dynamics: Exponential Convergence and Annealing
• Convex Analysis of the Mean Field Langevin Dynamics
• Mean-Field Limits for Two-Layer Neural Networks Trained with Consensus-Based Optimization