Lookahead Optimizer: k steps forward, 1 step back

2019/07/19 by Michael R. Zhang, James Lucas, Zhang, Michael R. +5 · 23 citations
Computer Science · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural and Evolutionary Computing (cs.NE) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.1907.08610

openalex publication_date 2019/07/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The vast majority of successful deep neural networks are trained using variants of stochastic gradient descent (SGD) algorithms. Recent attempts to improve SGD can be broadly categorized into two approaches: (1) adaptive learning rate schemes, such as AdaGrad and Adam, and (2) accelerated schemes, such as heavy-ball and Nesterov momentum. In this paper, we propose a new optimization algorithm, Lookahead, that is orthogonal to these previous approaches and iteratively updates two sets of weights. Intuitively, the algorithm chooses a search direction by looking ahead at the sequence of fast weights generated by another optimizer. We show that Lookahead improves the learning stability and lowers the variance of its inner optimizer with negligible computation and memory cost. We empirically demonstrate Lookahead can significantly improve the performance of SGD and Adam, even with their default hyperparameter settings on ImageNet, CIFAR-10/100, neural machine translation, and Penn Treebank.

Citations

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
• Averaging Weights Leads to Wider Optima and Better Generalization
• Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Smoothing DiLoCo with Primal Averaging for Faster Training of LLMs
• Greedy Alignment Principle for Optimizer Selection
• Beyond Adam: Disentangling Optimizer Effects in the Fine-Tuning of Atomistic Foundation Models
• Gradient Descent with Provably Tuned Learning-rate Schedules
• Unifying Sign and Magnitude for Optimizing Deep Vision Networks via ThermoLion
• HVAdam: A Full-Dimension Adaptive Optimizer
• WarpGAN: Warping-Guided 3D GAN Inversion with Style-Based Novel View Inpainting
• A Risk-Neutral Neural Operator for Arbitrage-Free SPX-VIX Term Structures
• Federated Learning via Meta-Variational Dropout
• Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning
• VidFace: A Full-Transformer Solver for Video FaceHallucination with Unaligned Tiny Snapshots
• Development of Deep Learning Optimizers: Approaches, Concepts, and Update Rules
• Generalization and Optimization of SGD with Lookahead
• Efficient Sharpness-aware Minimization for Improved Training of Neural Networks
• Beyond Individual Input for Deep Anomaly Detection on Tabular Data
• Potential of di-Higgs observation via a calibratable jet-free HH→ 4b framework
• FedEve: On Bridging the Client Drift and Period Drift for Cross-device Federated Learning
• Neural Scaling Laws Surpass Chemical Accuracy for the Many-Electron Schrödinger Equation
• Welcome New Doctor: Continual Learning with Expert Consultation and Autoregressive Inference for Whole Slide Image Analysis
• Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Models
• FedSWA: Improving Generalization in Federated Learning with Highly Heterogeneous Data via Momentum-Based Stochastic Controlled Weight Averaging
• The Price equation reveals a universal force-metric-bias law of algorithmic learning and natural selection
• Knowledge Generation -- Variational Bayes on Knowledge Graphs

Related

• Multilayer Lookahead: a Nested Version of Lookahead
• Optimization Methods in Deep Learning: A Comprehensive Overview
• Neumann Optimizer: A Practical Optimization Algorithm for Deep Neural Networks
• Frankenstein Optimizer: Harnessing the Potential by Revisiting Optimization Tricks
• Development of Deep Learning Optimizers: Approaches, Concepts, and Update Rules