Scalable and Practical Natural Gradient for Large-Scale Deep Learning

2020/02/13 by Kazuki Osawa, Yohei Tsuji, Osawa, Kazuki +9 · 1 citation
Computer Science · Mathematics · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Stochastic Gradient Optimization Techniques #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2002.06015

arXiv admin note: text overlap with arXiv:1811.12019

arxiv created 2020/02/13 · openalex publication_date 2020/02/13 · arxiv updated 2020/02/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large-scale distributed training of deep neural networks results in models with worse generalization performance as a result of the increase in the effective mini-batch size. Previous approaches attempt to address this problem by varying the learning rate and batch size over epochs and layers, or ad hoc modifications of batch normalization. We propose Scalable and Practical Natural Gradient Descent (SP-NGD), a principled approach for training models that allows them to attain similar generalization performance to models trained with first-order optimization methods, but with accelerated convergence. Furthermore, SP-NGD scales to large mini-batch sizes with a negligible computational overhead as compared to first-order methods. We evaluated SP-NGD on a benchmark task where highly optimized first-order methods are available as references: training a ResNet-50 model for image classification on ImageNet. We demonstrate convergence to a top-1 validation accuracy of 75.4% in 5.5 minutes using a mini-batch size of 32,768 with 1,024 GPUs, as well as an accuracy of 74.9% with an extremely large mini-batch size of 131,072 in 873 steps of SP-NGD.

Citations

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Natural Gradient Works Efficiently in Learning
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• L2 Regularization versus Batch and Weight Normalization
• Random Erasing Data Augmentation
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Large Batch Training of Convolutional Networks
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Don't Use Large Mini-Batches, Use Local SGD
• Fisher Information and Natural Gradient Learning of Random Deep Networks
• Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash
• On the interplay between noise and curvature and its effect on optimization and generalization
• Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networks

Cited by

• Beyond the Mean: Fisher-Orthogonal Projection for Natural Gradient Descent in Large Batch Training

Related

• Nonlinear Conjugate Gradients For Scaling Synchronous Distributed DNN Training
• Don't Use Large Mini-Batches, Use Local SGD
• DIVEBATCH: Accelerating Model Training Through Gradient-Diversity Aware Batch Size Adaptation
• Image Classification at Supercomputer Scale
• The Limit of the Batch Size