2020/10/25 by Zhiqi Bu, Bu, Zhiqi, Shiyun Xu +3 · 1 citation
Computer Science · Engineering · Physics and Astronomy · #Dynamical Systems (math.DS) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Model Reduction and Neural Networks #Neural Networks and Applications #Optimization and Control (math.OC) #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques
paper · pdf · doi:10.48550/arxiv.2010.13165
openalex publication_date 2020/10/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
When equipped with efficient optimization algorithms, the over-parameterized\nneural networks have demonstrated high level of performance even though the\nloss function is non-convex and non-smooth. While many works have been focusing\non understanding the loss dynamics by training neural networks with the\ngradient descent (GD), in this work, we consider a broad class of optimization\nalgorithms that are commonly used in practice. For example, we show from a\ndynamical system perspective that the Heavy Ball (HB) method can converge to\nglobal minimum on mean squared error (MSE) at a linear rate (similar to GD);\nhowever, the Nesterov accelerated gradient descent (NAG) may only converges to\nglobal minimum sublinearly.\n Our results rely on the connection between neural tangent kernel (NTK) and\nfinite over-parameterized neural networks with ReLU activation, which leads to\nanalyzing the limiting ordinary differential equations (ODE) for optimization\nalgorithms. We show that, optimizing the non-convex loss over the weights\ncorresponds to optimizing some strongly convex loss over the prediction error.\nAs a consequence, we can leverage the classical convex optimization theory to\nunderstand the convergence behavior of neural networks. We believe our approach\ncan also be extended to other optimization algorithms and network\narchitectures.\n