Gradient-based Hyperparameter Optimization through Reversible Learning

2015/02/11 by Dougal Maclaurin, David Duvenaud, Maclaurin, Dougal +3 · 1 voice · 42 citations
Computer Science · #Machine Learning and Data Classification #Gaussian Processes and Bayesian Inference #Advanced Neural Network Applications

paper · pdf · doi:10.48550/arxiv.1502.03492

Abstract

Tuning hyperparameters of learning algorithms is hard because gradients are usually unavailable. We compute exact gradients of cross-validation performance with respect to all hyperparameters by chaining derivatives backwards through the entire training procedure. These gradients allow us to optimize thousands of hyperparameters, including step-size and momentum schedules, weight initialization distributions, richly parameterized regularization schemes, and neural network architectures. We compute hyperparameter gradients by exactly reversing the dynamics of stochastic gradient descent with momentum.

Cited by

• Less Noise, More Signal: the DRR Effect for Better Optimizations of a Range of SE Tasks
• Automated Reinforcement Learning: An Overview
• DataRater: Meta-Learned Dataset Curation
• Bilevel gradient methods and the Morse parametric qualification condition
• Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration
• JKO-RAG: Distributional Retrieval as Wasserstein Free-Energy Gradient Flow
• Generative Bayesian Hyperparameter Tuning
• Greedy Alignment Principle for Optimizer Selection
• Neural Coherence : Find higher performance to out-of-distribution tasks from few samples
• Gradient Descent with Provably Tuned Learning-rate Schedules
• A Fully First-Order Layer for Differentiable Optimization
• Lower Complexity Bounds for Nonconvex-Strongly-Convex Bilevel Optimization with First-Order Oracles
• Bilevel Learning via Inexact Stochastic Gradient Descent
• Solving bilevel optimization via sequential minimax optimization
• A Review of Bilevel Optimization: Methods, Emerging Applications, and Recent Advancements
• First-Order Preconditioning via Hypergradient Descent
• Dynamic Automatic Differentiation of GPU Broadcast Kernels
• A Closer Look at Memorization in Deep Networks
• Semi-Amortized Variational Autoencoders
• Learning to Optimize Neural Nets
• Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability
• Bilevel Models for Adversarial Learning and A Case Study
• A Single-Loop First-Order Algorithm for Linearly Constrained Bilevel Optimization
• Scalable Principal-Agent Contract Design via Gradient-Based Optimization
• Computational Budget Should Be Considered in Data Selection
• End-to-End Test-Time Training for Long Context
• Bilevel optimization for learning hyperparameters: Application to solving PDEs and inverse problems with Gaussian processes
• Tricks from Deep Learning
• A Single-Loop Gradient Algorithm for Pessimistic Bilevel Optimization via Smooth Approximation
• Six Sigma For Neural Networks: Taguchi-based optimization
• An Introduction to Deep Learning for the Physical Layer
• Reinforcement Learning for Learning Rate Control
• Predict and Constrain: Modeling Cardinality in Deep Structured Prediction
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Learning to Teach with Dynamic Loss Functions
• Gradient-based Hyperparameter Optimization Over Long Horizons
• Interactive Agent Modeling by Learning to Probe
• Application of computer vision and deep learning models to automatically classify medically important mosquitoes in North Borneo, Malaysia
• Do Normalization Layers in a Deep ConvNet Really Need to Be Distinct?
• Flexible Dataset Distillation: Learn Labels Instead of Images
• Bilevel Programming for Hyperparameter Optimization and Meta-Learning
• Privileged Contrastive Pretraining for Multimodal Affect Modelling

Discussions

• Gradient-Based Hyperparameter Optimization Through Reversible Learning [hn, 58 points, 16 comments]

Related

• Gradient Descent: The Ultimate Optimizer
• Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters
• Gradient Descent with Provably Tuned Learning-rate Schedules
• Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions
• Optimizing Millions of Hyperparameters by Implicit Differentiation