ABC: Efficient Selection of Machine Learning Configuration on Large Dataset

2018/11/08 by Silu Huang, Chi Wang, Huang, Silu +5
Computer Science · #FOS: Computer and information sciences #Face and Expression Recognition #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.1811.03250

openalex publication_date 2018/11/08 · openalex created_date 2018/12/22 · openalex updated_date 2026/07/28

Abstract

A machine learning configuration refers to a combination of preprocessor, learner, and hyperparameters. Given a set of configurations and a large dataset randomly split into training and testing set, we study how to efficiently select the best configuration with approximately the highest testing accuracy when trained from the training set. To guarantee small accuracy loss, we develop a solution using confidence interval (CI)-based progressive sampling and pruning strategy. Compared to using full data to find the exact best configuration, our solution achieves more than two orders of magnitude speedup, while the returned top configuration has identical or close test accuracy.

Related

• Learning Multiple Defaults for Machine Learning Algorithms
• ExperienceThinking: Constrained Hyperparameter Optimization based on Knowledge and Pruning
• Hyperparameter Optimisation with Early Termination of Poor Performers
• Algorithm Configuration: Learning policies for the quick termination of\n poor performers
• Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets