Gradient-based Hyperparameter Optimization through Reversible Learning

2015/02/11 by Dougal Maclaurin, David Duvenaud, Maclaurin, Dougal +3 · 1 voice · 403 citations
Computer Science · Mathematics · #Advanced Neural Network Applications #Algorithm #Artificial intelligence #Artificial neural network #Chaining #Computer science #Gaussian Processes and Bayesian Inference #Gradient descent #Hyperparameter #Hyperparameter optimization #Initialization #Machine Learning and Data Classification #Machine learning #Mathematical optimization #Mathematics #Momentum (technical analysis) #Parameterized complexity #Regularization (linguistics) #Stochastic gradient descent #Support vector machine #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1502.03492

published in arXiv (Cornell University), 2113-2122 (Cornell University) · 10 figures. Submitted to ICML

openalex publication_date 2015/02/11 · arxiv created 2015/04/02 · arxiv updated 2015/04/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06

Abstract

Tuning hyperparameters of learning algorithms is hard because gradients are usually unavailable. We compute exact gradients of cross-validation performance with respect to all hyperparameters by chaining derivatives backwards through the entire training procedure. These gradients allow us to optimize thousands of hyperparameters, including step-size and momentum schedules, weight initialization distributions, richly parameterized regularization schemes, and neural network architectures. We compute hyperparameter gradients by exactly reversing the dynamics of stochastic gradient descent with momentum.

Cited by

• Less Noise, More Signal: the DRR Effect for Better Optimizations of a Range of SE Tasks
• Automated Reinforcement Learning: An Overview
• DataRater: Meta-Learned Dataset Curation
• Bilevel gradient methods and the Morse parametric qualification condition
• Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration
• JKO-RAG: Distributional Retrieval as Wasserstein Free-Energy Gradient Flow
• Generative Bayesian Hyperparameter Tuning
• Greedy Alignment Principle for Optimizer Selection
• Neural Coherence : Find higher performance to out-of-distribution tasks from few samples
• Gradient Descent with Provably Tuned Learning-rate Schedules
• A Fully First-Order Layer for Differentiable Optimization
• Lower Complexity Bounds for Nonconvex-Strongly-Convex Bilevel Optimization with First-Order Oracles
• Bilevel Learning via Inexact Stochastic Gradient Descent
• Solving bilevel optimization via sequential minimax optimization
• A Review of Bilevel Optimization: Methods, Emerging Applications, and Recent Advancements
• First-Order Preconditioning via Hypergradient Descent
• Dynamic Automatic Differentiation of GPU Broadcast Kernels
• A Closer Look at Memorization in Deep Networks
• Semi-Amortized Variational Autoencoders
• Learning to Optimize Neural Nets
• Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability
• Bilevel Models for Adversarial Learning and A Case Study
• A Single-Loop First-Order Algorithm for Linearly Constrained Bilevel Optimization
• Scalable Principal-Agent Contract Design via Gradient-Based Optimization
• Computational Budget Should Be Considered in Data Selection
• End-to-End Test-Time Training for Long Context
• Bilevel optimization for learning hyperparameters: Application to solving PDEs and inverse problems with Gaussian processes
• Tricks from Deep Learning
• A Single-Loop Gradient Algorithm for Pessimistic Bilevel Optimization via Smooth Approximation
• Six Sigma For Neural Networks: Taguchi-based optimization
• An Introduction to Deep Learning for the Physical Layer
• Reinforcement Learning for Learning Rate Control
• Predict and Constrain: Modeling Cardinality in Deep Structured Prediction
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Learning to Teach with Dynamic Loss Functions
• Gradient-based Hyperparameter Optimization Over Long Horizons
• Interactive Agent Modeling by Learning to Probe
• Application of computer vision and deep learning models to automatically classify medically important mosquitoes in North Borneo, Malaysia
• Do Normalization Layers in a Deep ConvNet Really Need to Be Distinct?
• Flexible Dataset Distillation: Learn Labels Instead of Images
• Towards Resilient Safety-driven Unlearning for Diffusion Models against Downstream Fine-tuning
• Bilevel Programming for Hyperparameter Optimization and Meta-Learning
• Privileged Contrastive Pretraining for Multimodal Affect Modelling
• Differentiable Rendering with Perturbed Optimizers
• Beyond Value Functions: Single-Loop Bilevel Optimization under Flatness Conditions
• Better Training Data Attribution via Better Inverse Hessian-Vector Products
• Differentiable Neural Input Search for Recommender Systems
• Stochastic Hyperparameter Optimization through Hypernetworks
• How to decay your learning rate
• Meta Approach to Data Augmentation Optimization
• Nonparametric Neural Networks
• A Comprehensive Overview and Survey of Recent Advances in Meta-Learning
• Discretization-free Knowledge Gradient Methods for Bayesian Optimization
• A statistical physics framework for optimal learning
• Neural Velocity for hyperparameter tuning
• Label-Free Long-Horizon 3D UAV Trajectory Prediction via Motion-Aligned RGB and Event Cues
• Hyperparameter Optimization: A Spectral Approach
• Cross-regularization: Adaptive Model Complexity through Validation Gradients
• BLUR: A Bi-Level Optimization Approach for LLM Unlearning
• Variational Inference using Implicit Distributions
• Model-Agnostic Meta-Learning using Runge-Kutta Methods
• Early Stopping is Nonparametric Variational Inference
• You Only Train Once
• Learning Discrete Structures for Graph Neural Networks
• Gradient-based Regularization Parameter Selection for Problems with Non-smooth Penalty Functions
• Instance Data Condensation for Image Super-Resolution
• Far-HO: A Bilevel Programming Package for Hyperparameter Optimization and Meta-Learning
• Programming with a Differentiable Forth Interpreter
• Structured Prediction Energy Networks
• Evolutionary Architecture Search For Deep Multitask Networks
• Representing smooth functions as compositions of near-identity functions with implications for deep network optimization
• POCAII: Parameter Optimization with Conscious Allocation using Iterative Intelligence
• DrMAD: Distilling Reverse-Mode Automatic Differentiation for Optimizing Hyperparameters of Deep Neural Networks
• Deep Q-Networks for Accelerating the Training of Deep Neural Networks
• Efficient Curvature-Aware Hypergradient Approximation for Bilevel Optimization
• Scalable Meta-Learning via Mixed-Mode Differentiation
• A Batched Scalable Multi-Objective Bayesian Optimization Algorithm
• End-to-End Learning for Structured Prediction Energy Networks
• Tuning Learning Rates with the Cumulative-Learning Constant
• Optimizing Hyperparameters in CNNs using Bilevel Programming in Time Series Data
• MAGIC: Near-Optimal Data Attribution for Deep Learning
• Diversity Networks: Neural Network Compression Using Determinantal Point\n Processes
• Optimizing Recurrent Neural Networks Architectures under Time Constraints
• An Adaptive Algorithm for Bilevel Optimization on Riemannian Manifolds

Discussions

• Gradient-Based Hyperparameter Optimization Through Reversible Learning [hn, 58 points, 16 comments]

Related

• Gradient Descent: The Ultimate Optimizer
• Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters
• Gradient Descent with Provably Tuned Learning-rate Schedules
• Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions
• Optimizing Millions of Hyperparameters by Implicit Differentiation