Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks

2019/05/27 by Boris Ginsburg, Patrice Castonguay, Ginsburg, Boris +15
Computer Science · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.1905.11286

openalex publication_date 2019/05/27 · openalex created_date 2019/05/29 · openalex updated_date 2026/07/28

Abstract

We propose NovoGrad, an adaptive stochastic gradient descent method with layer-wise gradient normalization and decoupled weight decay. In our experiments on neural networks for image classification, speech recognition, machine translation, and language modeling, it performs on par or better than well tuned SGD with momentum and Adam or AdamW. Additionally, NovoGrad (1) is robust to the choice of learning rate and weight initialization, (2) works well in a large batch setting, and (3) has two times smaller memory footprint than Adam.

Citations

• ImageNet Large Scale Visual Recognition Challenge
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Pointer Sentinel Mixture Models
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Block-Normalized Gradient Method: An Empirical Study for Training Deep Neural Network
• Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Mixed-Precision Training for NLP and Speech Recognition with OpenSeq2Seq
• Bag of Tricks for Image Classification with Convolutional Neural Networks
• Adaptive Gradient Methods with Dynamic Bound of Learning Rate

Related

• Reinforced stochastic gradient descent for deep neural network learning
• Stochastic Gradient Sampling for Enhancing Neural Networks Training
• Stochastic Gradient Descent: Going As Fast As Possible But Not Faster
• Block-Normalized Gradient Method: An Empirical Study for Training Deep Neural Network
• Adaptive Gradient Method with Resilience and Momentum