Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networks

2018/11/29 by Kazuki Osawa, Yohei Tsuji, Osawa, Kazuki +9 · 1 voice · 19 citations
Computer Science · Mathematics · #Advanced Neural Network Applications #Algorithm #Artificial intelligence #Benchmark (surveying) #Computer science #Convolutional neural network #Deep learning #Deep neural networks #Domain Adaptation and Few-Shot Learning #Generalization #Kronecker delta #Machine Learning and ELM #Mathematics #Normalization (sociology) #Scale (ratio) #cs.CV #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1811.12019

published in arXiv (Cornell University) (Cornell University) · 10 pages, 7 figures. Accepted at CVPR 2019, Long Beach, CA

openalex publication_date 2018/11/29 · arxiv created 2019/03/30 · arxiv updated 2019/04/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Large-scale distributed training of deep neural networks suffer from the generalization gap caused by the increase in the effective mini-batch size. Previous approaches try to solve this problem by varying the learning rate and batch size over epochs and layers, or some ad hoc modification of the batch normalization. We propose an alternative approach using a second-order optimization method that shows similar generalization capability to first-order methods, but converges faster and can handle larger mini-batches. To test our method on a benchmark where highly optimized first-order methods are available as references, we train ResNet-50 on ImageNet. We converged to 75% Top-1 validation accuracy in 35 epochs for mini-batch sizes under 16,384, and achieved 75% even with a mini-batch size of 131,072, which took only 978 iterations.

Citations

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Natural Gradient Works Efficiently in Learning
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• L2 Regularization versus Batch and Weight Normalization
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks
• Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes

Cited by

• Tractable structured natural gradient descent using local parameterizations
• Optimization for deep learning: theory and algorithms
• Fast Approximation of the Gauss-Newton Hessian Matrix for the Multilayer Perceptron
• Augment your batch: better training with larger batches
• Scalable and Practical Natural Gradient for Large-Scale Deep Learning

Discussions

• Training ResNet-50 on ImageNet in 35 Epochs Using Second-Order Optimization [hn, 49 points, 10 comments]

Related

• Convolutional Neural Network Training with Distributed K-FAC
• Scalable and Practical Natural Gradient for Large-Scale Deep Learning
• Study on the Large Batch Size Training of Neural Networks Based on the Second Order Gradient
• Nonlinear Conjugate Gradients For Scaling Synchronous Distributed DNN Training
• Data optimization for large batch distributed training of deep neural networks