An overview of gradient descent optimization algorithms

2016/09/15 by Sebastian Ruder, Ruder, Sebastian · 3 voices · 235 citations
Computer Science · #Stochastic Gradient Optimization Techniques #Medical Image Segmentation Techniques #Face and Expression Recognition

paper · pdf · doi:10.48550/arxiv.1609.04747

Abstract

Gradient descent optimization algorithms, while increasingly popular, are often used as black-box optimizers, as practical explanations of their strengths and weaknesses are hard to come by. This article aims to provide the reader with intuitions with regard to the behaviour of different algorithms that will allow her to put them to use. In the course of this overview, we look at different variants of gradient descent, summarize challenges, introduce the most common optimization algorithms, review architectures in a parallel and distributed setting, and investigate additional strategies for optimizing gradient descent.

Citations

• ADADELTA: An Adaptive Learning Rate Method
• A Stochastic Approximation Method

Cited by

• Efficient optimisation of multi-parameter quantum control protocols for strongly-coupled systems
• Shesha: Opportunistic In-network Acceleration of Asynchronous Distributed Reinforcement Learning
• Material-agnostic temperature field prediction for metal additive manufacturing via a parametric PINN framework
• A Gradient Free Neural Network Framework Based on Universal\n Approximation Theorem
• GRAFFL: Gradient-free Federated Learning of a Bayesian Generative Model
• Evaluation of Neural Networks for Image Recognition Applications: Designing a 0-1 MILP Model of a CNN to create adversarials
• Temporal Autoencoder with U-Net Style Skip-Connections for Frame\n Prediction
• Training Neural Networks with an algorithm for piecewise linear\n functions
• MUSCLE: Strengthening Semi-Supervised Learning Via Concurrent Unsupervised Learning Using Mutual Information Maximization
• Large-Scale Deep Learning Optimizations: A Comprehensive Survey
• On the Equivalence of Neural and Production Networks
• Universal Pansharpening Model
• Domain Decomposition of Large Neural Network Surrogate Models
• Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools
• Deep Learning Based Regression and Multi-class Models for Acute Oral Toxicity Prediction with Automatic Chemical Feature Extraction
• Debate-Enhanced Pseudo Labeling and Frequency-Aware Progressive Debiasing for Weakly-Supervised Camouflaged Object Detection with Scribble Annotations
• OLR-WA: Online Weighted Average Linear Regression in Multivariate Data Streams
• An Additively Preconditioned Trust Region Strategy for Machine Learning
• Optimizing the Adversarial Perturbation with a Momentum-based Adaptive Matrix
• OLR-WAA: Adaptive and Drift-Resilient Online Regression with Dynamic Weighted Averaging
• Learning Dynamics in Memristor-Based Equilibrium Propagation
• Open Horizons: Evaluating Deep Models in the Wild
• Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injections
• Accelerating gradient descent and Adam via fractional gradients
• Learnability Window in Gated Recurrent Neural Networks
• Constraint-oriented biased quantum search for linear constrained combinatorial optimization problems
• Parameter efficient hybrid spiking-quantum convolutional neural network with surrogate gradient and quantum data-reupload
• Self-organized Operational Neural Networks with Generative Neurons
• Temporal mixture ensemble models for intraday volume forecasting in cryptocurrency exchange markets
• Detecting Problem Statements in Peer Assessments
• Deep Filament Extraction for 3D Concrete Printing
• Physics-informed neural networks method in high-dimensional integrable systems
• Gradient Descent Algorithm Survey
• Learning Scalable Temporal Representations in Spiking Neural Networks Without Labels
• Multiple Learning for Regression in big data
• Deep Learning Framework for Enhanced Neutrino Reconstruction of Single-line Events in the ANTARES Telescope
• Quantum measurement tomography with mini-batch stochastic gradient descent
• Trimaximal Mixing Patterns Meet the First JUNO Result
• NTK-Guided Implicit Neural Teaching
• Towards Evolutionary Optimization Using the Ising Model
• Simultaneous Localization and 3D-Semi Dense Mapping for Micro Drones Using Monocular Camera and Inertial Sensors
• Deep Neural Network Based Resource Allocation for V2X Communications
• Deep Learning in Wide-field Surveys: Fast Analysis of Strong Lenses in Ground-based Cosmic Experiments
• The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous\n Neural Networks
• Multi-Loss Sub-Ensembles for Accurate Classification with Uncertainty Estimation
• Neograd: Near-Ideal Gradient Descent
• Dynamic Temperature Scheduler for Knowledge Distillation
• FineSkiing: A Fine-grained Benchmark for Skiing Action Quality Assessment
• A Novel Stochastic Stratified Average Gradient Method: Convergence Rate and Its Complexity
• Fidelity sweet spot in transmon qubit rings under strong connectivity noise
• Hybrid Quantum-Classical Graph Convolutional Network
• FLEX: Continuous Agent Evolution via Forward Learning from Experience
• Neural Machine Translation and Sequence-to-sequence Models: A Tutorial
• Variational noise mitigation in quantum circuits: the case of Quantum Fourier Transform
• ODE approximation for the Adam algorithm: General and overparametrized setting
• Unified Theory of Adaptive Variance Reduction
• DORAEMON: A Unified Library for Visual Object Modeling and Representation Learning at Scale
• Shellular Metamaterial Design via Compact Electric Potential Parametrization
• CLAX: Fast and Flexible Neural Click Models in JAX
• Constrained Performance Boosting Control for Nonlinear Systems via ADMM
• Predictive Auxiliary Learning for Belief-based Multi-Agent Systems
• Extended dynamic mode decomposition with dictionary learning: A data-driven adaptive spectral decomposition of the Koopman operator
• Gradient Descent as Loss Landscape Navigation: a Normative Framework for Deriving Learning Rules
• On Higher-order Moments in Adam
• Recent Advances in Recurrent Neural Networks
• A Tutorial on Deep Learning for Music Information Retrieval
• From center to surrounding: An interactive learning framework for hyperspectral image classification
• Deep learning for pedestrians: backpropagation in CNNs
• Robust and Active Learning for Deep Neural Network Regression
• Modulating Regularization Frequency for Efficient Compression-Aware Model Training
• MMCoVaR: Multimodal COVID-19 Vaccine Focused Data Repository for Fake News Detection and a Baseline Architecture for Classification
• Adaptive unified contrastive learning with graph-based feature aggregator for imbalanced medical image classification
• Implementation of Parallel Simplified Swarm Optimization in CUDA
• Teaching Machine Learning to Software Engineers
• Compressing Heavy-Tailed Weight Matrices for Non-Vacuous Generalization Bounds
• Decision-based Universal Adversarial Attack
• Spatio-temporal masked autoencoder-based phonetic segments classification from ultrasound
• ZORB: A Derivative-Free Backpropagation Algorithm for Neural Networks
• Clickbait Detection in Tweets Using Self-attentive Network
• Topology Optimization under Uncertainty using a Stochastic Gradient-based Approach
• Adaptive Online Learning with Momentum for Contingency-based Voltage Stability Assessment
• An improvement of the convergence proof of the ADAM-Optimizer
• Flexible Operator Embeddings via Deep Learning
• Seismic vulnerability modelling of building portfolios using artificial neural networks
• Dynamically Weighted Momentum with Adaptive Step Sizes for Efficient Deep Network Training
• Sparse Network Inversion for Key Instance Detection in Multiple Instance Learning
• Cross-Domain Adaptation for Animal Pose Estimation
• Variable Projected Augmented Lagrangian Methods for Generalized Lasso Problems
• Bayesian Optimization Algorithms for Accelerator Physics
• Robust Non-negative Proximal Gradient Algorithm for Inverse Problems
• Benchmarking VQE Configurations: Architectures, Initializations, and Optimizers for Silicon Ground State Energy
• Learning Reconfigurable Representations for Multimodal Federated Learning with Missing Data
• What Do We Understand About Convolutional Networks?
• Joint Design of Radar Waveform and Detector via End-to-end Learning with Waveform Constraints
• xMem: A CPU-Based Approach for Accurate Estimation of GPU Memory in Deep Learning Training Workloads
• DeepLocalize: Fault Localization for Deep Neural Networks
• Semantic World Models
• QCFace: Image Quality Control for boosting Face Representation & Recognition
• An Encode-then-Decompose Approach to Unsupervised Time Series Anomaly Detection on Contaminated Training Data--Extended Version
• Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout
• Changes in United States Summer Temperatures Revealed by Explainable Neural Networks
• Influence Functions in Deep Learning Are Fragile
• Fighter: Unveiling the Graph Convolutional Nature of Transformers in Time Series Modeling
• Double Neural Counterfactual Regret Minimization
• PoTS: Proof-of-Training-Steps for Backdoor Detection in Large Language Models
• Feature Selection and Regularization in Multi-Class Classification: An Empirical Study of One-vs-Rest Logistic Regression with Gradient Descent Optimization and L1 Sparsity Constraints
• Hyper-Parameter Optimization: A Review of Algorithms and Applications
• A Stochastic Algorithm for Searching Saddle Points with Convergence Guarantee
• Using Kolmogorov-Smirnov Distance for Measuring Distribution Shift in Machine Learning
• Learning to Recognize Correctly Completed Procedure Steps in Egocentric Assembly Videos through Spatio-Temporal Modeling
• DeepTrust: Multi-Step Classification through Dissimilar Adversarial Representations for Robust Android Malware Detection
• Sinkformers: Transformers with Doubly Stochastic Attention
• Comparative Explanations via Counterfactual Reasoning in Recommendations
• A Blockchain‐based Cyber Attack Detection Scheme for Decentralized Internet of Things using Software‐Defined Network
• Learning Dynamics of VLM Finetuning
• Hyperlink Regression via Bregman Divergence
• Stochastic Probabilistic Programs
• Distributed Learning of Deep Neural Networks using Independent Subnet Training
• The impact of the additional features on the performance of regression analysis: a case study on regression analysis of music signal
• Analytical Survey of Learning with Low-Resource Data: From Analysis to Investigation
• Memory Retrieval and Consolidation in Large Language Models through Function Tokens
• SSGD: A safe and efficient method of gradient descent
• The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
• TreeNet: Layered Decision Ensembles
• QDeepGR4J: Quantile-based ensemble of deep learning and GR4J hybrid rainfall-runoff models for extreme flow prediction with uncertainty quantification
• Image Generation Based on Image Style Extraction
• CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMs
• AI-assisted Advanced Propellant Development for Electric Propulsion
• Neural Networks as Functional Classifiers
• Soft-Di[M]O: Improving One-Step Discrete Image Generation with Soft Embeddings
• Communication-Efficient and Interoperable Distributed Learning
• Effective continuous equations for adaptive SGD: a stochastic analysis view
• Learning-Based Collaborative Control for Bi-Manual Tactile-Reactive Grasping
• Supervised Contrastive Learning
• Sobolev acceleration for neural networks
• Improved Frequency Tracking with Adaptive Moments for Narrowband Interference Mitigation in GNSS
• Improve Single-Point Zeroth-Order Optimization Using High-Pass and Low-Pass Filters
• Why to "grow" and "harvest" deep learning models?
• A Performance Comparison of Loss Functions for Deep Face Recognition
• A Combined Data-driven and Physics-driven Method for Steady Heat Conduction Prediction using Deep Convolutional Neural Networks
• Adaptive Elastic Training for Sparse Deep Learning on Heterogeneous Multi-GPU Servers
• MYSTIKO : : Cloud-Mediated, Private, Federated Gradient Descent
• A Variant of Gradient Descent Algorithm Based on Gradient Averaging
• Development of Deep Learning Optimizers: Approaches, Concepts, and Update Rules
• Neuromodulated Learning in Deep Neural Networks
• Neural Network Based Framework for Passive Intermodulation Cancellation in MIMO Systems
• The Root Finding Problem Revisited: Beyond the Robbins-Monro procedure
• Incorporating Visual Cortical Lateral Connection Properties into CNN: Recurrent Activation and Excitatory-Inhibitory Separation
• Exploring the Relationship between Brain Hemisphere States and Frequency Bands through Deep Learning Optimization Techniques
• Solving Zero-Sum Games through Alternating Projections
• An Analysis of Optimizer Choice on Energy Efficiency and Performance in Neural Network Training
• Inferring Soil Drydown Behaviour with Adaptive Bayesian Online Changepoint Analysis
• EmbeddedML: A New Optimized and Fast Machine Learning Library
• AdaSGD: Bridging the gap between SGD and Adam
• Privacy-preserving Federated Bayesian Learning of a Generative Model for Imbalanced Classification of Clinical Data
• A survey on deep learning approaches for breast cancer diagnosis
• On Faster Convergence of Scaled Sign Gradient Descent
• SugarcaneShuffleNet: A Very Fast, Lightweight Convolutional Neural Network for Diagnosis of 15 Sugarcane Leaf Diseases
• Survey: Machine Learning in Production Rendering
• A Capsule-unified Framework of Deep Neural Networks for Graphical Programming
• Packing Sparse Convolutional Neural Networks for Efficient Systolic Array Implementations: Column Combining Under Joint Optimization
• A machine learning framework for data driven acceleration of computations of differential equations
• Heart Disease Prediction: A Comparative Study of Optimisers Performance in Deep Neural Networks
• Decentor-V: Lightweight ML Training on Low-Power RISC-V Edge Devices
• Two-Stage Framework for Efficient UAV-Based Wildfire Video Analysis with Adaptive Compression and Fire Source Detection
• Finding the right scale of a network: Efficient identification of causal\n emergence through spectral clustering
• Uncertainty-Aware Neural Networks for Fuzzy Dark Matter Model Selection from \texorpdfstringx_{\rm HI}x_HI Measurements
• Deep Gamblers: Learning to Abstain with Portfolio Theory
• 1D convolutional neural networks and applications: A survey
• Neural Architecture Search via Bregman Iterations
• Event Detection and Classification for Long Range Sensing of Elephants Using Seismic Signal
• Application of Decision Rules for Handling Class Imbalance in Semantic Segmentation
• Learning Fair Scoring Functions: Bipartite Ranking under ROC-based Fairness Constraints
• A constrained optimization approach to nonlinear system identification through simulation error minimization
• Learning by training: emergent return-point memory from cyclically tuning disordered sphere packings
• CascadeFormer: A Family of Two-stage Cascading Transformers for Skeleton-based Human Action Recognition
• Deep Learning in Mobile and Wireless Networking: A Survey
• Theory Foundation of Physics-Enhanced Residual Learning
• Adaptive Heavy-Tailed Stochastic Gradient Descent
• Learning compact generalizable neural representations supporting perceptual grouping
• SincQDR-VAD: A Noise-Robust Voice Activity Detection Framework Leveraging Learnable Filters and Ranking-Aware Optimization
• GRADSTOP: Early Stopping of Gradient Descent via Posterior Sampling
• A Local Block Coordinate Descent Algorithm for the Convolutional Sparse Coding Model
• A Deep Learning Application for Psoriasis Detection
• Adaptive Hierarchical Hyper-gradient Descent
• From ANN to BNN: Inferring Reionization Parameters using Uncertainty-aware Emulators of 21-cm Summaries
• MuSACo: Multimodal Subject-Specific Selection and Adaptation for Expression Recognition with Co-Training
• Time-Scale Coupling Between States and Parameters in Recurrent Neural Networks
• Deep Learning based Estimation of Weaving Target Maneuvers
• Learning to See: Convolutional Neural Networks for the Analysis of Social Science Data
• Quaternion Collaborative Filtering for Recommendation
• Selectivity correction with online machine learning
• Stochastic Gradient Langevin Dynamics Algorithms with Adaptive Drifts
• Machine Biometrics -- Towards Identifying Machines in a Smart City Environment
• NSGA-Net: Neural Architecture Search using Multi-Objective Genetic Algorithm
• QAMRO: Quality-aware Adaptive Margin Ranking Optimization for Human-aligned Assessment of Audio Generation Systems
• A Comprehensive Survey of Multilingual Neural Machine Translation
• GrCAN: Gradient Boost Convolutional Autoencoder with Neural Decision Forest
• Representation Understanding via Activation Maximization
• IWA: Integrated Gradient based White-box Attacks for Fooling Deep Neural Networks
• Parameters for the best convergence of an optimization algorithm On-The-Fly
• DeePore: A deep learning workflow for rapid and comprehensive characterization of porous materials
• Neural Network Training via Stochastic Alternating Minimization with Trainable Step Sizes
• Unsupervised Pairwise Learning Optimization Framework for Cross-Corpus EEG-Based Emotion Recognition Based on Prototype Representation
• Training Deep Neural Networks via Branch-and-Bound
• DualNet: Locate Then Detect Effective Payload with Deep Attention Network
• Stress-Aware Resilient Neural Training
• Personalized Dynamic Treatment Regimes in Continuous Time: A Bayesian Approach for Optimizing Clinical Decisions with Timing
• Optimizing Convergence for Iterative Learning of ARIMA for Stationary Time Series
• Demystifying Learning Rate Policies for High Accuracy Training of Deep Neural Networks
• WEEP: A Differentiable Nonconvex Sparse Regularizer via Weakly-Convex Envelope
• Communication-Efficient Distributed Training for Collaborative Flat Optima Recovery in Deep Learning
• Layerwise Optimization by Gradient Decomposition for Continual Learning
• Fast Stochastic Variance Reduced Gradient Method with Momentum Acceleration for Machine Learning
• Almost fault-tolerant quantum machine learning with drastic overhead reduction
• Machine Learning based Radio Environment Map Estimation for Indoor Visible Light Communication
• Explicit Gradient Learning
• Deep learning empowers genomic selection of pest-resistant grapevine
• Machine learning in geo- and environmental sciences: From small to large scale
• Activated Gradients for Deep Neural Networks
• On The State of Data In Computer Vision: Human Annotations Remain Indispensable for Developing Deep Learning Models
• Demystifying BERT: Implications for Accelerator Design
• BigSurvSGD: Big Survival Data Analysis via Stochastic Gradient Descent
• Low Dimensional Landscape Hypothesis is True: DNNs can be Trained in Tiny Subspaces
• A Multi-modal and Multi-task Learning Method for Action Unit and Expression Recognition
• A Pragmatic AI Approach to Creating Artistic Visual Variations by Neural Style Transfer
• Sparse-View Spectral CT Reconstruction Using Deep Learning
• 2nd-order Updates with 1st-order Complexity
• On Deep Learning for Radio Resource Management in A Non-stationary Radio Environment
• Expression Recognition Analysis in the Wild
• OptTyper: Probabilistic Type Inference by Optimising Logical and Natural\n Constraints
• Memory-Efficient Factorization Machines via Binarizing both Data and Model Coefficients
• Lily-like twist distribution in toroidal nematics
• A recurrent multi-scale approach to RBG-D Object Recognition
• Learning rate [wikipedia]

Discussions

• An overview of gradient descent optimization algorithms [hn, 78 points, 8 comments]
• An overview of gradient descent optimization algorithms (2017) [hn, 1 points, 0 comments]
• "An overview of gradient descent optimization algorithms" by Sebastian Ruder 📄 arxiv.org/abs/1609.04747 A must-read for anyone diving into optimization techniques! [bsky, 0 points, 0 comments]

Related

• Gradient Descent Algorithm Survey
• A Comprehensive Study on Optimization Strategies for Gradient Descent In Deep Learning
• Gradient Descent, Stochastic Optimization, and Other Tales
• Enhancing Deep Learning with Optimized Gradient Descent: Bridging Numerical Methods and Neural Network Training
• Optimization using Parallel Gradient Evaluations on Multiple Parameters