Optimizing Millions of Hyperparameters by Implicit Differentiation

2019/11/06 by Lorraine, Jonathan, Vicol, Paul, Duvenaud, David · 15 citations
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.1911.02590

Abstract

We propose an algorithm for inexpensive gradient-based hyperparameter optimization that combines the implicit function theorem (IFT) with efficient inverse Hessian approximations. We present results about the relationship between the IFT and differentiating through optimization, motivating our algorithm. We use the proposed approach to train modern network architectures with millions of weights and millions of hyper-parameters. For example, we learn a data-augmentation network - where every weight is a hyperparameter tuned for validation performance - outputting augmented training examples. Jointly tuning weights and hyperparameters with our approach is only a few times more costly in memory and compute than standard training.

Cited by

• Learning Lévy density via adaptive RKHS regression with bi-level optimization
• Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration
• Gradient Descent with Provably Tuned Learning-rate Schedules
• A Fully First-Order Layer for Differentiable Optimization
• Modality Alignment across Trees on Heterogeneous Hyperbolic Manifolds
• A flexible and differentiable coil proxy for stellarator equilibrium optimization
• Scalable Principal-Agent Contract Design via Gradient-Based Optimization
• Computational Budget Should Be Considered in Data Selection
• Bilevel optimization for learning hyperparameters: Application to solving PDEs and inverse problems with Gaussian processes
• A Single-Loop Gradient Algorithm for Pessimistic Bilevel Optimization via Smooth Approximation
• Taught Well Learned Ill: Towards Distillation-conditional Backdoor Attack
• Information-Theoretic Bayesian Optimization for Bilevel Optimization Problems
• Reversible Deep Equilibrium Models
• Curvature Learning for Generalization of Hyperbolic Neural Networks
• Fusion Sampling Validation in Data Partitioning for Machine Learning

Related

• Scalable One-Pass Optimisation of High-Dimensional Weight-Update Hyperparameters by Implicit Differentiation
• Gradient-based Hyperparameter Optimization through Reversible Learning
• Implicit differentiation for fast hyperparameter selection in non-smooth convex learning
• Gradient-based Hyperparameter Optimization Over Long Horizons
• Implicit differentiation of Lasso-type models for hyperparameter optimization