Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization

2016/03/21 by Lisha Li, Kevin Jamieson, Giulia DeSalvo +2 · 1 voice · 20 citations
Computer Science · Mathematics · #cs.LG #stat.ML

paper · pdf

published as Journal of Machine Learning Research 18 (2018) 1-52 · Changes: - Updated to JMLR version

arxiv published 2016/03/21 · arxiv created 2018/06/18 · arxiv updated 2018/06/20

Abstract

Performance of machine learning algorithms depends critically on identifying a good set of hyperparameters. While recent approaches use Bayesian optimization to adaptively select configurations, we focus on speeding up random search through adaptive resource allocation and early-stopping. We formulate hyperparameter optimization as a pure-exploration non-stochastic infinite-armed bandit problem where a predefined resource like iterations, data samples, or features is allocated to randomly sampled configurations. We introduce a novel algorithm, Hyperband, for this framework and analyze its theoretical properties, providing several desirable guarantees. Furthermore, we compare Hyperband with popular Bayesian optimization methods on a suite of hyperparameter optimization problems. We observe that Hyperband can provide over an order-of-magnitude speedup over our competitor set on a variety of deep-learning and kernel-based learning problems.

Cited by

• AIRA₂: Overcoming Bottlenecks in AI Research Agents
• Relatron: Automating Relational Machine Learning over Relational Databases
• LLM-Guided ANN Index Optimization for Human-Object Interaction Retrieval
• Tuning Learning Rates with the Cumulative-Learning Constant
• SCOPE-MRI: Bankart Lesion Detection as a Case Study in Data Curation and Deep Learning for Challenging Diagnoses
• Modular Machine Learning: An Indispensable Path towards New-Generation Large Language Models
• HyperController: A Hyperparameter Controller for Fast and Stable Training of Reinforcement Learning Neural Networks
• Biologically-Grounded Multi-Encoder Architectures as Developability Oracles for Antibody Design
• GRIMIP: A General Framework for Instance-Specific Configuration of MIP Solvers Using LLMs
• Preconditioning Natural and Second Order Gradient Descent in Quantum Optimization: A Performance Benchmark
• CAPO: Cost-Aware Prompt Optimization
• VeLU: Variance-enhanced Learning Unit for Deep Neural Networks
• Some Optimizers are More Equal: Understanding the Role of Optimizers in Group Fairness
• A computational framework for longitudinal medication adherence prediction in breast cancer survivors: A social cognitive theory based approach
• Code generation for solving and differentiating through convex optimization problems
• Machine learning in flight parameter-based structural load prediction: A review and framework proposal
• Relay, Don't Route: Adaptive Population Handoff for Cost-Efficient LLM-Driven Evolution
• Evaluating Machine Learning Models for Post-Wildfire Debris-Flow Prediction
• Never Start from Scratch: Expediting On-Device LLM Personalization via Explainable Model Selection
• Frozen Layers: Memory-efficient Many-fidelity Hyperparameter Optimization

Discussions

• Everything Is ddmin araffin.github.io/post/hyperpa... Hyperband arxiv.org/abs/1603.06560 [bsky, 2 points, 0 comments]

Related

• Combination of Hyperband and Bayesian Optimization for Hyperparameter Optimization in Deep Learning
• BOHB: Robust and Efficient Hyperparameter Optimization at Scale
• HyperArm Bandit Optimization: A Novel approach to Hyperparameter Optimization and an Analysis of Bandit Algorithms in Stochastic and Adversarial Settings
• An Asymptotically Optimal Multi-Armed Bandit Algorithm and Hyperparameter Optimization
• Hyperparameter Optimization in Machine Learning