On Large-Batch Training for Deep Learning: Generalization Gap and Sharp\n Minima

2016/09/15 by Nitish Shirish Keskar, Dheevatsa Mudigere, Keskar, Nitish Shirish +7 · 185 citations
Computer Science · #Stochastic Gradient Optimization Techniques #Machine Learning and Algorithms #Machine Learning and ELM

paper · pdf · doi:10.48550/arxiv.1609.04836

Abstract

The stochastic gradient descent (SGD) method and its variants are algorithms\nof choice for many Deep Learning tasks. These methods operate in a small-batch\nregime wherein a fraction of the training data, say 32-512 data points, is\nsampled to compute an approximation to the gradient. It has been observed in\npractice that when using a larger batch there is a degradation in the quality\nof the model, as measured by its ability to generalize. We investigate the\ncause for this generalization drop in the large-batch regime and present\nnumerical evidence that supports the view that large-batch methods tend to\nconverge to sharp minimizers of the training and testing functions - and as is\nwell known, sharp minima lead to poorer generalization. In contrast,\nsmall-batch methods consistently converge to flat minimizers, and our\nexperiments support a commonly held view that this is due to the inherent noise\nin the gradient estimation. We discuss several strategies to attempt to help\nlarge-batch methods eliminate this generalization gap.\n

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• The Loss Surfaces of Multilayer Networks
• Sample size selection in optimization methods for machine learning
• Gradient-based learning applied to document recognition
• Gradient Descent Converges to Minimizers

Cited by

• Geometry-Aware Optimization for Respiratory Sound Classification: Enhancing Sensitivity with SAM-Optimized Audio Spectrogram Transformers
• SPRKD: Effective Knowledge Distillation for Deep Neural Networks via Saddle Region Approximation
• Mini-batch Noise Lowers Sharpness via Dominant-Subspace Fluctuations
• Semantic Space Search Trajectory Networks
• Are Flat Minima an Illusion?
• The Interaction Bottleneck of Deep Neural Networks: Discovery, Proof, and Modulation
• When Does Learning Renormalize? Sufficient Conditions for Power Law Spectral Dynamics
• Sharpness-aware Federated Graph Learning
• Mitigating Hallucinations in Healthcare LLMs with Granular Fact-Checking and Domain-Specific Adaptation
• SALVE: Sparse Autoencoder-Latent Vector Editing for Mechanistic Control of Neural Networks
• OLR-WA: Online Weighted Average Linear Regression in Multivariate Data Streams
• KD-PINN: Knowledge-Distilled PINNs for ultra-low-latency real-time neural PDE solvers
• CurvaDion: Curvature-Adaptive Distributed Orthonormalization
• Large and Small Model Collaboration for Air Interface
• Neural Network-based Partial-Linear Single-Index Models for Environmental Mixtures Analysis
• TPV: Parameter Perturbations Through the Lens of Test Prediction Variance
• Weakly Supervised Tuberculosis Localization in Chest X-rays through Knowledge Distillation
• Renormalizable Spectral-Shell Dynamics as the Origin of Neural Scaling Laws
• Robustness of Probabilistic Models to Low-Quality Data: A Multi-Perspective Analysis
• Scaling Behavior of Discrete Diffusion Language Models
• Minimizing Layerwise Activation Norm Improves Generalization in Federated Learning
• A Bootstrap Perspective on Stochastic Gradient Descent
• Entropic Confinement and Mode Connectivity in Overparameterized Neural Networks
• Data Curation Through the Lens of Spectral Dynamics: Static Limits, Dynamic Acceleration, and Practical Oracles
• Cross-Geometry Transfer Learning in Fast Electromagnetic Shower Simulation
• Do Flat Minima Improve Sparse Novel View Synthesis?
• A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Bias
• Fast LLM Post-training via Decoupled and Fastest-of-N Speculation
• BackdoorBench: A Comprehensive Benchmark and Analysis of Backdoor Learning
• Weight Variance Amplifier Improves Accuracy in High-Sparsity One-Shot Pruning
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentation
• Don't Decay the Learning Rate, Increase the Batch Size
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• A Progressive Batching L-BFGS Method for Machine Learning
• Zero-Order Sharpness-Aware Minimization
• DPlis: Boosting Utility of Differentially Private Deep Learning via Randomized Smoothing
• Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians
• Robustness to Pruning Predicts Generalization in Deep Neural Networks
• Forecasting Thermospheric Density with Transformers for Multi-Satellite Orbit Management
• Sharp Minima Can Generalize: A Loss Landscape Perspective On Data
• Linear Mode Connectivity under Data Shifts for Deep Ensembles of Image Classifiers
• Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?
• Flat Minima and Generalization: Insights from Stochastic Convex Optimization
• The Generalization-Stability Tradeoff In Neural Network Pruning
• SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systems
• The Curvature Rate λ: A Scalar Measure of Input-Space Sharpness in Neural Networks
• Bayesian Natural Gradient Fine-Tuning of CLIP Models via Kalman Filtering
• A Framework Based on Graph Cellular Automata for Similarity Evaluation in Urban Spatial Networks
• Information-Theoretic Greedy Layer-wise Training for Traffic Sign Recognition
• DP-FedPGN: Finding Global Flat Minima for Differentially Private Federated Learning via Penalizing Gradient Norm
• Gradient Descent as Loss Landscape Navigation: a Normative Framework for Deriving Learning Rules
• Gradient-Weight Alignment as a Train-Time Proxy for Generalization in Classification Tasks
• A Convexity-dependent Two-Phase Training Algorithm for Deep Neural Networks
• An Empirical Study of Example Forgetting during Deep Neural Network Learning
• On improving deep learning generalization with adaptive sparse connectivity
• Extrapolation for Large-batch Training in Deep Learning
• Loss Landscape Dependent Self-Adjusting Learning Rates in Decentralized Stochastic Gradient Descent
• Deep learning generalizes because the parameter-function map is biased towards simple functions
• Where is the Information in a Deep Neural Network?
• Shape Matters: Understanding the Implicit Bias of the Noise Covariance
• On Scale-out Deep Learning Training for Cloud and HPC
• Towards Better Generalization: BP-SVRG in Training Deep Neural Networks
• Analytic Characterization of the Hessian in Shallow ReLU Models: A Tale of Symmetry
• A Closer Look at Memorization in Deep Networks
• Compressing Heavy-Tailed Weight Matrices for Non-Vacuous Generalization Bounds
• Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
• Stochastic Training of Residual Networks: a Differential Equation\n Viewpoint
• Noise Aggregation Analysis Driven by Small-Noise Injection: Efficient Membership Inference for Diffusion Models
• On Surprising Effectiveness of Masking Updates in Adaptive Optimizers
• Sharpness-Guided Group Relative Policy Optimization via Probability Shaping
• Modality-Aware SAM: Sharpness-Aware-Minimization Driven Gradient Modulation for Harmonized Multimodal Learning
• Seeing Through the Brain: New Insights from Decoding Visual Stimuli with fMRI
• A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization
• More Than Memory Savings: Zeroth-Order Optimization Mitigates Forgetting in Continual Learning
• Convergence Analysis of SGD under Expected Smoothness
• Mini-batch stochastic gradient descent with dynamic sample sizes
• A Unified Perspective on Optimization in Machine Learning and Neuroscience: From Gradient Descent to Neural Adaptation
• Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout
• Large batch size training of neural networks with adversarial training and second-order information
• Position: Many generalization measures for deep learning are fragile
• Stochastic Difference-of-Convex Optimization with Momentum
• When Flatness Does (Not) Guarantee Adversarial Robustness
• Flatness-Aware Stochastic Gradient Langevin Dynamics
• LinearSR: Unlocking Linear Attention for Stable and Efficient Image Super-Resolution
• DYNAMIX: RL-based Adaptive Batch Size Optimization in Distributed Machine Learning Systems
• Stronger generalization bounds for deep nets via a compression approach
• Unveiling the Power of Multiple Gossip Steps: A Stability-Based Generalization Analysis in Decentralized Training
• Rényi Sharpness: A Novel Sharpness that Strongly Correlates with Generalization
• Local SGD Converges Fast and Communicates Little
• Adjusting Initial Noise to Mitigate Memorization in Text-to-Image Diffusion Models
• The Effect of Label Noise on the Information Content of Neural Representations
• How does the optimizer implicitly bias the model merging loss landscape?
• Categorical Invariants of Learning Dynamics
• Optimal Scaling Needs Optimal Norm
• Adaptively Sampling-Reusing-Mixing Decomposed Gradients to Speed Up Sharpness Aware Minimization
• SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticality
• Maximizing Parallelism in Distributed Training for Huge Neural Networks
• Reconcile Certified Robustness and Accuracy for DNN-based Smoothed Majority Vote Classifier
• Sharpness of Minima in Deep Matrix Factorization: Exact Expressions
• How Does Preconditioning Guide Feature Learning in Deep Neural Networks?
• Hybrid Dual-Batch and Cyclic Progressive Learning for Efficient Distributed Training
• Beyond Magic Words: Sharpness-Aware Prompt Evolving for Robust Large Language Models with TARE
• Dynamics of Learning: Generative Schedules from Latent ODEs
• TRUST: Test-Time Refinement using Uncertainty-Guided SSM Traverses
• Understanding Generalization in Deep Learning via Tensor Methods
• Fine-tuning Done Right in Model Editing
• Sharpness-Aware Minimization Can Hallucinate Minimizers
• Laplacian Smoothing Gradient Descent
• A Unified Noise-Curvature View of Loss of Trainability
• Towards joint scaling laws with optimal batch size schedules
• Algebraic Approach to Ridge-Regularized Mean Squared Error Minimization in Minimal ReLU Neural Network
• Parallel Blockwise Knowledge Distillation for Deep Neural Network Compression
• Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noise
• Tangent Space Separability in Feedforward Neural Networks
• Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokking
• Graph Coloring for Multi-Task Learning
• Neural Network Based Framework for Passive Intermodulation Cancellation in MIMO Systems
• MEC-Quant: Maximum Entropy Coding for Extremely Low Bit Quantization-Aware Training
• DIVEBATCH: Accelerating Model Training Through Gradient-Diversity Aware Batch Size Adaptation
• Nonparametric Online Learning Using Lipschitz Regularized Deep Neural Networks
• On the Convergence of SARAH and Beyond
• Understanding Batch Normalization
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• The Lifecycle Principle: Stabilizing Dynamic Neural Networks with State Memory
• Gradient Diversity: a Key Ingredient for Scalable Distributed Learning
• Experimental validation result for OASIS software
• On the Escaping Efficiency of Distributed Adversarial Training Algorithms
• On the geometry of flat minima
• Lyapunov stability of the Euler method
• Balanced Sharpness-Aware Minimization for Imbalanced Regression
• Momentum-based variance-reduced proximal stochastic gradient method for composite nonconvex stochastic optimization
• Model compression via distillation and quantization
• How Far Are We from True Unlearnability?
• Nonlinear Collaborative Scheme for Deep Neural Networks
• Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margin
• Explaining How Quantization Disparately Skews a Model
• On Using Large-Batches in Federated Learning
• Efficient Sharpness-aware Minimization for Improved Training of Neural Networks
• CoT-Space: A Theoretical Framework for Internal Slow-Thinking via Reinforcement Learning
• Enhancing Gradient Variance and Differential Privacy in Quantum Federated Learning
• Variation-matching sensitivity-based virtual fields for hyperelastic material model calibration
• LSAM: Asynchronous Distributed Training with Landscape-Smoothed Sharpness-Aware Minimization
• VASSO: Variance Suppression for Sharpness-Aware Minimization
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• Why flatness does and does not correlate with generalization for deep neural networks
• Adaptive Heavy-Tailed Stochastic Gradient Descent
• MERIT: Maximum-normalized Element-wise Ratio for Language Model Large-batch Training
• Decoupled Greedy Learning of CNNs
• A Convex Neural Network Solver for DCOPF with Generalization Guarantees
• C-Flat++: Towards a More Efficient and Powerful Framework for Continual Learning
• Flatness-aware Curriculum Learning via Adversarial Difficulty
• Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Tasks
• Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach
• WISCA: A Lightweight Model Transition Method to Improve LLM Training via Weight Scaling
• Robustness to Augmentations as a Generalization metric
• Wormhole Dynamics in Deep Neural Networks
• Twin-Boot: Uncertainty-Aware Optimization via Online Two-Sample Bootstrapping
• Inter-Class Relational Loss for Small Object Detection: A Case Study on License Plates
• Large Batch Training Does Not Need Warmup
• FLAIR: Frequency- and Locality-Aware Implicit Neural Representations
• Beyond the Mean: Fisher-Orthogonal Projection for Natural Gradient Descent in Large Batch Training
• Training Machine Learning Models on Human Spatio-temporal Mobility Data: An Experimental Study [Experiment Paper]
• Optimal Condition for Initialization Variance in Deep Neural Networks: An SGD Dynamics Perspective
• Using Mode Connectivity for Loss Landscape Analysis
• Contrastive Weight Regularization for Large Minibatch SGD
• Semi-Supervised Learning Enabled by Multiscale Deep Neural Network Inversion
• Subsampling Factorization Machine Annealing
• Multi-level Collaborative Distillation Meets Global Workspace Model: A Unified Framework for OCIL
• Statistical Theory of Multi-stage Newton Iteration Algorithm for Online Continual Learning
• Tractable Sharpness-Aware Learning of Probabilistic Circuits
• Sentiment-Aware Stock Price Prediction with Transformer and LLM-Generated Formulaic Alpha
• Sensitivity of Stability: Theoretical & Empirical Analysis of Replicability for Adaptive Data Selection in Transfer Learning
• Arithmetic addition of two integers by deep image classification networks: experiments to quantify their autonomous reasoning ability
• Multiple Sclerosis Lesion Segmentation from Brain MRI via Fully Convolutional Neural Networks
• Superior resilience to poisoning and amenability to unlearning in quantum machine learning
• Communication-efficient distributed SGD with Sketching
• Noisy Gradient Descent Converges to Flat Minima for Nonconvex Matrix Factorization
• Communication-Efficient Distributed Training for Collaborative Flat Optima Recovery in Deep Learning
• What training reveals about neural network complexity
• Irredundant k-Fold Cross-Validation
• The Effectiveness of Multitask Learning for Phenotyping with Electronic\n Health Records Data
• The Price equation reveals a universal force-metric-bias law of algorithmic learning and natural selection
• Improved Techniques For Weakly-Supervised Object Localization
• Activated Gradients for Deep Neural Networks

Related

• On the Generalization Benefit of Noise in Stochastic Gradient Descent
• Towards Theoretical Understanding of Large Batch Training in Stochastic Gradient Descent
• The Effect of Network Width on the Performance of Large-batch Training
• A Bayesian Perspective on Generalization and Stochastic Gradient Descent
• Special Properties of Gradient Descent with Large Learning Rates