Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization

2021/08/25 by Difan Zou, Yuan Cao, Zou, Difan +5 · 6 citations
Computer Science · Mathematics · #Artificial intelligence #Artificial neural network #Computer science #FOS: Computer and information sciences #FOS: Mathematics #Generalization #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Machine learning #Mathematical analysis #Mathematics #Neural Networks and Applications #Optimization and Control (math.OC) #Regularization (linguistics) #Stochastic Gradient Optimization Techniques #cs.LG #math.OC #stat.ML

paper · pdf · doi:10.48550/arxiv.2108.11371

published in arXiv (Cornell University) (Cornell University) · 42 pages, 2 figures and 1 table

arxiv created 2021/08/25 · openalex publication_date 2021/08/25 · arxiv updated 2021/08/26 · openalex created_date 2021/08/30 · openalex updated_date 2026/07/28

Abstract

Adaptive gradient methods such as Adam have gained increasing popularity in deep learning optimization. However, it has been observed that compared with (stochastic) gradient descent, Adam can converge to a different solution with a significantly worse test error in many deep learning applications such as image classification, even with a fine-tuned regularization. In this paper, we provide a theoretical explanation for this phenomenon: we show that in the nonconvex setting of learning over-parameterized two-layer convolutional neural networks starting from the same random initialization, for a class of data distributions (inspired from image data), Adam and gradient descent (GD) can converge to different global solutions of the training objective with provably different generalization errors, even with weight decay regularization. In contrast, we show that if the training objective is convex, and the weight decay regularization is employed, any optimization algorithms including Adam and GD will converge to the same solution if the training is successful. This suggests that the inferior generalization performance of Adam is fundamentally tied to the nonconvex landscape of deep learning optimization.

Citations

• A Convergence Theory for Deep Learning via Over-Parameterization
• Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networks

Cited by

• Implicit Bias of Per-sample Adam on Separable Data: Departure from the Full-batch Regime
• Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noise
• The Rich and the Simple: On the Implicit Bias of Adam and SGD
• On the Role of Label Noise in the Feature Learning Process
• Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)
• Gradient Descent Robustly Learns the Intrinsic Dimension of Data in Training Convolutional Neural Networks

Related

• Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks
• The Rich and the Simple: On the Implicit Bias of Adam and SGD
• Normalized Direction-preserving Adam
• Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Acceleration
• On the Convergence of Adam and Beyond