Quadratic number of nodes is sufficient to learn a dataset via gradient descent

2019/11/13 by B. Krishna Das, Das, Biswarup, Eugène Golikov +1
Computer Science · #68T20 #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning and ELM #Neural Networks and Applications #Optimization and Control (math.OC) #Statistics Theory (math.ST) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.1911.05402

openalex publication_date 2019/11/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We prove that if an activation function satisfies some mild conditions and number of neurons in a two-layered fully connected neural network with this activation function is beyond a certain threshold, then gradient descent on quadratic loss function finds the optimal weights of input layer for global minima in linear time. This threshold value is an improvement over previously obtained values. We hypothesise that this bound cannot be improved by the method we are using in this work.

Citations

• Gradient Descent Finds Global Minima for Generalizable Deep Neural Networks of Practical Sizes

Related

• Convergence of gradient descent for deep neural networks
• Avoiding Spurious Local Minima in Deep Quadratic Networks
• On the convergence of gradient descent for two layer neural networks
• Stationary Points of a Shallow Neural Network with Quadratic Activations and the Global Optimality of the Gradient Descent Algorithm
• Gradient Descent on Infinitely Wide Neural Networks: Global Convergence and Generalization