Gradient-based Hyperparameter Optimization Over Long Horizons

2020/07/15 by Paul Micaelli, Amos Storkey, Micaelli, Paul +1
Computer Science · Physics and Astronomy · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification #Model Reduction and Neural Networks

paper · pdf · doi:10.48550/arxiv.2007.07869

openalex publication_date 2020/07/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Gradient-based hyperparameter optimization has earned a widespread popularity in the context of few-shot meta-learning, but remains broadly impractical for tasks with long horizons (many gradient steps), due to memory scaling and gradient degradation issues. A common workaround is to learn hyperparameters online, but this introduces greediness which comes with a significant performance drop. We propose forward-mode differentiation with sharing (FDS), a simple and efficient algorithm which tackles memory scaling issues with forward-mode differentiation, and gradient degradation issues by sharing hyperparameters that are contiguous in time. We provide theoretical guarantees about the noise reduction properties of our algorithm, and demonstrate its efficiency empirically by differentiating through ∼ 10⁴ gradient steps of unrolled optimization. We consider large hyperparameter search ranges on CIFAR-10 where we significantly outperform greedy gradient-based alternatives, while achieving × 20 speedups compared to the state-of-the-art black-box methods. Code is available at: \urlhttps://github.com/polo5/FDS

Citations

• Scalable Bayesian Optimization Using Deep Neural Networks
• On the difficulty of training Recurrent Neural Networks
• Gradient-based Hyperparameter Optimization through Reversible Learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• DARTS: Differentiable Architecture Search
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Meta-Learning with Implicit Gradients
• PyTorch: An Imperative Style, High-Performance Deep Learning Library

Related

• Gradient-based Hyperparameter Optimization through Reversible Learning
• Gradient Descent: The Ultimate Optimizer
• Optimizing Millions of Hyperparameters by Implicit Differentiation
• Online Hyperparameter Meta-Learning with Hypergradient Distillation
• Understanding the Mechanisms of Fast Hyperparameter Transfer