Optimizing Millions of Hyperparameters by Implicit Differentiation

2019/11/06 by Jonathan Lorraine, Paul Vicol, Lorraine, Jonathan +3 · 33 citations
Computer Science · Mathematics · #Advanced Neural Network Applications #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1911.02590

Submitted to AISTATS 2020

arxiv created 2019/11/06 · openalex publication_date 2019/11/06 · arxiv updated 2019/11/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose an algorithm for inexpensive gradient-based hyperparameter optimization that combines the implicit function theorem (IFT) with efficient inverse Hessian approximations. We present results about the relationship between the IFT and differentiating through optimization, motivating our algorithm. We use the proposed approach to train modern network architectures with millions of weights and millions of hyper-parameters. For example, we learn a data-augmentation network - where every weight is a hyperparameter tuned for validation performance - outputting augmented training examples. Jointly tuning weights and hyperparameters with our approach is only a few times more costly in memory and compute than standard training.

Citations

• Sur les opérations dans les ensembles abstraits et leur application aux équations intégrales
• Variational Dropout and the Local Reparameterization Trick
• Long Short-Term Memory
• Gradient-based learning applied to document recognition
• Practical Bayesian Optimization of Machine Learning Algorithms
• ImageNet classification with deep convolutional neural networks
• Categorical Reparameterization with Gumbel-Softmax
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Regularizing and Optimizing LSTM Language Models
• Regularization for Deep Learning: A Taxonomy
• Stochastic Hyperparameter Optimization through Hypernetworks
• Understanding Short-Horizon Bias in Stochastic Meta-Optimization
• ScreenerNet: Learning Self-Paced Curriculum for Deep Neural Networks
• On First-Order Meta-Learning Algorithms
• Parallel Architecture and Hyperparameter Search via Successive Halving and Classification
• AutoAugment: Learning Augmentation Policies from Data
• Stable Opponent Shaping in Differentiable Games
• AutoAssist: A Framework to Accelerate Training of Deep Neural Networks
• Adversarial Learning of General Transformations for Data Augmentation
• DrMAD: Distilling Reverse-Mode Automatic Differentiation for Optimizing Hyperparameters of Deep Neural Networks
• Meta-Learning with Implicit Gradients
• Generalized Inner Loop Meta-Learning
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Learning Lévy density via adaptive RKHS regression with bi-level optimization
• Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration
• Gradient Descent with Provably Tuned Learning-rate Schedules
• A Fully First-Order Layer for Differentiable Optimization
• Modality Alignment across Trees on Heterogeneous Hyperbolic Manifolds
• A flexible and differentiable coil proxy for stellarator equilibrium optimization
• Scalable Principal-Agent Contract Design via Gradient-Based Optimization
• Computational Budget Should Be Considered in Data Selection
• Bilevel optimization for learning hyperparameters: Application to solving PDEs and inverse problems with Gaussian processes
• A Single-Loop Gradient Algorithm for Pessimistic Bilevel Optimization via Smooth Approximation
• Taught Well Learned Ill: Towards Distillation-conditional Backdoor Attack
• Information-Theoretic Bayesian Optimization for Bilevel Optimization Problems
• Reversible Deep Equilibrium Models
• Curvature Learning for Generalization of Hyperbolic Neural Networks
• Fusion Sampling Validation in Data Partitioning for Machine Learning
• Better Training Data Attribution via Better Inverse Hessian-Vector Products
• Hypergradient-based Bilevel Reinforcement Learning with Improved Sample Complexity
• Robust Molecular Property Prediction via Densifying Scarce Labeled Data
• Score Distillation Sampling for Audio: Source Separation, Synthesis, and Beyond
• A Hybrid Subgradient Method for Nonsmooth Nonconvex Bilevel Optimization
• Rethinking Data Mixture for Large Language Models: A Comprehensive Survey and New Perspectives
• Bi-Level Policy Optimization with Nyström Hypergradients
• POCAII: Parameter Optimization with Conscious Allocation using Iterative Intelligence
• Argus: Federated Non-convex Bilevel Learning over 6G Space-Air-Ground Integrated Network
• Meta-Learning Symmetries by Reparameterization
• You Only Look One Step: Accelerating Backpropagation in Diffusion Sampling with Gradient Shortcuts
• A Provably Convergent Plug-and-Play Framework for Stochastic Bilevel Optimization
• Scalable Meta-Learning via Mixed-Mode Differentiation
• Diff2SP: Diffusion Models for Correlated Scenario Generation in Stochastic Programming
• MiniMax Learning of Interpretable Factored Stochastic Policies from Conjoint Data, with Uncertainty Quantification
• Perturbed Gradient Descent via Convex Quadratic Approximation for Nonconvex Bilevel Optimization
• Frozen Layers: Memory-efficient Many-fidelity Hyperparameter Optimization
• An Adaptive Algorithm for Bilevel Optimization on Riemannian Manifolds

Related

• Scalable One-Pass Optimisation of High-Dimensional Weight-Update Hyperparameters by Implicit Differentiation
• Gradient-based Hyperparameter Optimization through Reversible Learning
• Implicit differentiation for fast hyperparameter selection in non-smooth convex learning
• Gradient-based Hyperparameter Optimization Over Long Horizons
• Implicit differentiation of Lasso-type models for hyperparameter optimization