Asymptotic convexity of wide and shallow neural networks

2025/06/23 by Vivek S. Borkar, Borkar, Vivek, Parthe Pandit +1
Computer Science · #68T07 #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Neural Networks and Applications #Probability (math.PR) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2507.01044

openalex publication_date 2025/06/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

For a simple model of shallow and wide neural networks, we show that the epigraph of its input-output map as a function of the network parameters approximates epigraph of a. convex function in a precise sense. This leads to a plausible explanation of their observed good performance.

Citations

• Wide Neural Networks Forget Less Catastrophically
• Do Wider Neural Networks Really Help Adversarial Robustness?
• Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networks
• Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• The Expressive Power of Neural Networks: A View from the Width
• The loss surface of deep and wide neural networks

Related

• Why Learning of Large-Scale Neural Networks Behaves Like Convex Optimization
• Convex Formulation of Overparameterized Deep Neural Networks
• On the Convex Behavior of Deep Neural Networks in Relation to the Layers' Width
• The Curious Case of Convex Neural Networks
• Convexification of Neural Graph