Learned Optimizers that Scale and Generalize

2017/03/14 by Olga Wichrowska, Wichrowska, Olga, Niru Maheswaranathan +11 · 15 citations
Computer Science · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Multimodal Machine Learning Applications #Neural and Evolutionary Computing (cs.NE)

paper · pdf · doi:10.48550/arxiv.1703.04813

openalex publication_date 2017/03/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Learning to learn has emerged as an important direction for achieving artificial intelligence. Two of the primary barriers to its adoption are an inability to scale to larger problems and a limited ability to generalize to new tasks. We introduce a learned gradient descent optimizer that generalizes well to new tasks, and which has significantly reduced memory and computation overhead. We achieve this by introducing a novel hierarchical RNN architecture, with minimal per-parameter overhead, augmented with additional architectural features that mirror the known structure of optimization tasks. We also develop a meta-training ensemble of small, diverse optimization tasks capturing common properties of loss landscapes. The optimizer learns to outperform RMSProp/ADAM on problems in this corpus. More importantly, it performs comparably or better when applied to small convolutional neural networks, despite seeing no neural networks in its meta-training set. Finally, it generalizes to train Inception V3 and ResNet V2 architectures on the ImageNet dataset for thousands of steps, optimization problems that are of a vastly different scale than those it was trained on. We release an open source implementation of the meta-training algorithm.

Citations

• Building machines that learn and think like people
• Learning to reinforcement learn
• Neural Architecture Search with Reinforcement Learning

Cited by

• NeurGO: Learning to Generate Elite Candidates for Meta-Black-Box Expensive Optimization
• Evolving Deep Learning Optimizers
• Greedy Alignment Principle for Optimizer Selection
• Neural Coherence : Find higher performance to out-of-distribution tasks from few samples
• Transformer-based Scalable Beamforming Optimization via Deep Residual Learning
• Learning Regularizers: Learning Optimizers that can Regularize
• Learning a Zeroth-Order Optimizer for Fine-Tuning LLMs
• FunL2O: LLM-Guided Feature Function Design for Learning to Optimize
• A General Decoupled Learning Framework for Parameterized Image Operators
• How to Train Your MAML to Excel in Few-Shot Classification
• Adaptive Hierarchical Hyper-gradient Descent
• L-SR1: Learned Symmetric-Rank-One Preconditioning
• Meta Continual Learning
• Learning to Optimize: A Primer and A Benchmark
• Learning from Oblivion: Predicting Knowledge Overflowed Weights via Retrodiction of Forgetting

Related

• Tasks, stability, architecture, and compute: Training more effective\n learned optimizers, and using them to train themselves
• Narrowing the Focus: Learned Optimizers for Pretrained Models
• Celo: Training Versatile Learned Optimizers on a Compute Diet
• A Generalizable Approach to Learning Optimizers
• Understanding and correcting pathologies in the training of learned optimizers