Spurious Local Minima of Deep ReLU Neural Networks in the Neural Tangent Kernel Regime

2018/06/13 by Nitta, Tohru
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.1806.04884

Abstract

In this paper, we theoretically prove that the deep ReLU neural networks do not lie in spurious local minima in the loss landscape under the Neural Tangent Kernel (NTK) regime, that is, in the gradient descent training dynamics of the deep ReLU neural networks whose parameters are initialized by a normal distribution in the limit as the widths of the hidden layers tend to infinity.

Related

• Finite Depth and Width Corrections to the Neural Tangent Kernel
• Spurious Local Minima are Common in Two-Layer ReLU Neural Networks
• On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widths
• Revisiting the Neural Tangent Kernel: the role of large width and depth
• Small nonlinearities in activation functions create bad local minima in neural networks