2023/01/30 by Killamsetty, Krishnateja, Alexandre Evfimievski, Evfimievski, Alexandre V. +8 · 1 citation
Computer Science · #Machine Learning and Data Classification #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning
paper · pdf · doi:10.48550/arxiv.2301.13287
Training deep networks and tuning hyperparameters on large datasets is computationally intensive. One of the primary research directions for efficient training is to reduce training costs by selecting well-generalizable subsets of training data. Compared to simple adaptive random subset selection baselines, existing intelligent subset selection approaches are not competitive due to the time-consuming subset selection step, which involves computing model-dependent gradients and feature embeddings and applies greedy maximization of submodular objectives. Our key insight is that removing the reliance on downstream model parameters enables subset selection as a pre-processing step and enables one to train multiple models at no additional cost. In this work, we propose MILO, a model-agnostic subset selection framework that decouples the subset selection from model training while enabling superior model convergence and performance by using an easy-to-hard curriculum. Our empirical results indicate that MILO can train models 3× - 10 × faster and tune hyperparameters 20× - 75 × faster than full-dataset training or tuning without compromising performance.