Revisiting Small Batch Training for Deep Neural Networks

2018/04/20 by Dominic Masters, Masters, Dominic, Carlo Luschi +1 · 4 voices · 12 citations
Computer Science · Mathematics · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #cs.CV #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1804.07612

Abstract

Modern deep neural network training is typically based on mini-batch stochastic gradient optimization. While the use of large mini-batches increases the available computational parallelism, small batch training has been shown to provide improved generalization performance and allows a significantly smaller memory footprint, which might also be exploited to improve machine throughput. In this paper, we review common assumptions on learning rate scaling and training duration, as a basis for an experimental comparison of test performance for different mini-batch sizes. We adopt a learning rate that corresponds to a constant average weight update per gradient calculation (i.e., per unit cost of computation), and point out that this results in a variance of the weight updates that increases linearly with the mini-batch size m. The collected experimental results for the CIFAR-10, CIFAR-100 and ImageNet datasets show that increasing the mini-batch size progressively reduces the range of learning rates that provide stable convergence and acceptable test performance. On the other hand, small mini-batch sizes provide more up-to-date gradient calculations, which yields more stable and reliable training. The best performance has been consistently obtained for mini-batch sizes between m = 2 and m = 32, which contrasts with recent work advocating the use of mini-batch sizes in the thousands.

Cited by

• Transfer Learning for Analysis of Collective and Non-Collective Thomson Scattering Spectra
• Scaling Behavior of Discrete Diffusion Language Models
• DYNAMIX: RL-based Adaptive Batch Size Optimization in Distributed Machine Learning Systems
• Hybrid Layer-Wise ANN-SNN With Surrogate Spike Encoding-Decoding Structure
• Characterizing the Performance of Accelerated Jetson Edge Devices for Training Deep Learning Models
• DIVEBATCH: Accelerating Model Training Through Gradient-Diversity Aware Batch Size Adaptation
• Chest X-ray Pneumothorax Segmentation Using EfficientNet-B4 Transfer Learning in a U-Net Architecture
• Rethinking Safety in LLM Fine-tuning: An Optimization Perspective
• Sentiment-Aware Stock Price Prediction with Transformer and LLM-Generated Formulaic Alpha
• EMA Without the Lag: Bias-Corrected Iterate Averaging Schemes
• On the Importance of Fundamental Properties in Quantum-Classical Machine Learning Models
• Operator-based machine learning framework for generalizable prediction of unsteady treatment dynamics in stormwater infrastructure

Discussions

• Revisiting Small Batch Training for Deep Neural Networks [hn, 3 points, 0 comments]
• Revisiting Small Batch Training for Deep Neural Networks [hn, 2 points, 0 comments]
• Revisiting Small Batch Training for Deep NNs; batch_size less than 64 is best [hn, 1 points, 0 comments]
• Revisiting Small Batch Training for Deep Neural Networks [hn, 1 points, 0 comments]

Related

• AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks
• Increasing Both Batch Size and Learning Rate Accelerates Stochastic Gradient Descent
• Can we learn better with hard samples?
• Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Training
• The Effect of Network Width on the Performance of Large-batch Training