Cautious Optimizers: Improving Training with One Line of Code

2024/11/25 by Kaizhao Liang, Lizhang Chen, Liang, Kaizhao +5 · 12 voices · 11 citations
#cs.LG #cs.AI #cs.CL #cs.CV #cs.DM

paper · pdf · doi:10.48550/arxiv.2411.16085

Abstract

AdamW has been the default optimizer for transformer pretraining. For many years, our community searched for faster and more stable optimizers with only constrained positive outcomes. In this work, we propose a one-line modification in Pytorch to any momentum-based optimizer, which we rename cautious optimizer, e.g. C-AdamW and C-Lion. Our theoretical result shows that this modification preserves Adam's Hamiltonian function and it does not break the convergence guarantee under the Lyapunov analysis. In addition, a whole new family of optimizers is revealed by our theoretical insight. Among them, we pick the simplest one for empirical experiments, showing not only consistent speed-up on LLM pretraining, but also image classification, with minimum extra tuning on hyperparameters. Code is available at https://github.com/kyleliang919/C-Optim.

Cited by

• Physics-informed Neural Operators for Predicting 3D Electromagnetic Fields Transformed by Metasurfaces
• Greedy Alignment Principle for Optimizer Selection
• Algorithms Trained on Normal Chest X-rays Can Predict Health Insurance Types
• On Surprising Effectiveness of Masking Updates in Adaptive Optimizers
• What Really Matters in Matrix-Whitening Optimizers?
• Relative Scaling Laws for LLMs
• Cautious Weight Decay
• Development of Deep Learning Optimizers: Approaches, Concepts, and Update Rules
• On the Convergence of Muon and Beyond
• ViewSparsifier: Killing Redundancy in Multi-View Plant Phenotyping
• Training Dynamics of the Cooldown Stage in Warmup-Stable-Decay Learning Rate Scheduler

Discussions

• Cautious Optimizers: Improving Training with One Line of Code [hn, 66 points, 2 comments]
• New academic life goal: writing a paper about one line of code (that actually does something useful). "Cautious Optimizers: Improving Training with One Line of Code" arxiv.org/abs/2411.16085 Adding [bsky, 2 points, 0 comments]
• *Cautious Optimizers: Improving Training with One Line of Code* by Liang et al. Adding a simple masking operation to momentum-based optimizers can significantly boost their speed. arxiv.org/abs/2411 [bsky, 2 points, 0 comments]
• Is there any interesting work on those things? Except replicating them in small scales? Do we know anything about momentum across stages of training? Why do we see spikes? (Maybe related to this? arxi [bsky, 1 points, 1 comments]
• Cautious Optimizers: Improving Training with One Line of Code [hn, 1 points, 1 comments]
• Cautious Optimizers: Improving Training with One Line of Code https://arxiv.org/abs/2411.16085 [comments] [43 points] [telegram] [bsky, 0 points, 0 comments]
• Cautious Optimizers: Improving Training with One Line of Code https://arxiv.org/abs/2411.16085 (https://news.ycombinator.com/item?id=43244666) [bsky, 0 points, 0 comments]
• New paper on Optimizers: This paper introduces a single-line modification to momentum-based optimizers, termed the Cautious Optimizer. Empirical experiments with Llama and MAE pretraining demonstrate [bsky, 0 points, 0 comments]
• Cautious Optimizers: Improving Training with One Line of Code https://arxiv.org/abs/2411.16085 (https://news.ycombinator.com/item?id=43244666) [bsky, 0 points, 0 comments]
• Not a new paper, but I hadn't seen it til now arxiv.org/abs/2411.16085 [bsky, 0 points, 0 comments]
• Cautious Optimizers: Improving Training with One Line of Code [bsky, 0 points, 0 comments]
• Cautious Optimizers: Improving Training with One Line of Code https://arxiv.org/abs/2411.16085 [bsky, 0 points, 0 comments]

Related

• AdamHD: Decoupled Huber Decay Regularization for Language Model Pre-Training
• PyLO: Towards Accessible Learned Optimizers in PyTorch
• An Isometric Stochastic Optimizer
• Taming Transformer Without Using Learning Rate Warmup
• Should I try multiple optimizers when fine-tuning pre-trained Transformers for NLP tasks? Should I tune their hyperparameters?