Sub-Optimal Local Minima Exist for Neural Networks with Almost All Non-Linear Activations

2019/11/04 by Tian Ding, Dawei Li, Ding, Tian +3
Computer Science · #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Neural Networks and Applications #Optimization and Control (math.OC) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.1911.01413

openalex publication_date 2019/11/04 · openalex created_date 2020/11/23 · openalex updated_date 2026/07/28

Abstract

Does over-parameterization eliminate sub-optimal local minima for neural networks? An affirmative answer was given by a classical result in [59] for 1-hidden-layer wide neural networks. A few recent works have extended the setting to multi-layer neural networks, but none of them has proved every local minimum is global. Why is this result never extended to deep networks? In this paper, we show that the task is impossible because the original result for 1-hidden-layer network in [59] can not hold. More specifically, we prove that for any multi-layer network with generic input data and non-linear activation functions, sub-optimal local minima can exist, no matter how wide the network is (as long as the last hidden layer has at least two neurons). While the result of [59] assumes sigmoid activation, our counter-example covers a large set of activation functions (dense in the set of continuous functions), indicating that the limitation is not due to the specific activation. Our result indicates that "no bad local-min" may be unable to explain the benefit of over-parameterization for training neural nets.

Citations

• Depth Creates No Bad Local Minima
• An Analytical Formula of Population Gradient for two-layered ReLU\n network and its Applications in Convergence and Critical Point Analysis
• Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networks
• Recovery Guarantees for One-hidden-layer Neural Networks
• Porcupine Neural Networks: (Almost) All Local Optima are Global
• Understanding the Loss Surface of Neural Networks for Binary Classification
• A Convergence Theory for Deep Learning via Over-Parameterization
• On the Stability Properties and the Optimization Landscape of Training\n Problems with Squared Loss for Neural Networks and General Nonlinear Conic\n Approximation Schemes

Related

• Small nonlinearities in activation functions create bad local minima in neural networks
• Non-attracting Regions of Local Minima in Deep and Wide Neural Networks
• The loss surface of deep and wide neural networks
• Deep linear neural networks with arbitrary loss: All local minima are global
• No bad local minima: Data independent training error guarantees for\n multilayer neural networks