On the convergence of gradient descent for two layer neural networks

2019/09/30 by Lei Li, Li, Lei
Computer Science · #30E10 #60B10 #68W25 #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Neural Networks and Applications #Optimization and Control (math.OC) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.1909.13671

openalex publication_date 2019/09/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

It has been shown that gradient descent can yield the zero training loss in the over-parametrized regime (the width of the neural networks is much larger than the number of data points). In this work, combining the ideas of some existing works, we investigate the gradient descent method for training two-layer neural networks for approximating some target continuous functions. By making use the generic chaining technique from probability theory, we show that gradient descent can yield an exponential convergence rate, while the width of the neural networks needed is independent of the size of the training data. The result also implies some strong approximation ability of the two-layer neural networks without curse of dimensionality.

Citations

• Learning Overparameterized Neural Networks via Stochastic Gradient\n Descent on Structured Data

Related

• Gradient Descent on Infinitely Wide Neural Networks: Global Convergence and Generalization
• Approximation and Gradient Descent Training with Neural Networks
• Stochastic Gradient Descent for Two-layer Neural Networks
• Generalization Guarantees of Gradient Descent for Multi-Layer Neural Networks
• Leveraging the two timescale regime to demonstrate convergence of neural networks