Coresets via Bilevel Optimization for Continual Learning and Streaming

2020/06/06 by Zalán Borsos, Borsos, Zalán, Mojmír Mutný +3 · 40 citations
Computer Science · Engineering · Mathematics · #Domain Adaptation and Few-Shot Learning #Machine Learning and Algorithms #Sparse and Compressive Sensing Techniques #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2006.03875

NeurIPS 2020

arxiv created 2020/10/22 · arxiv updated 2020/10/23

Abstract

Coresets are small data summaries that are sufficient for model training. They can be maintained online, enabling efficient handling of large data streams under resource constraints. However, existing constructions are limited to simple models such as k-means and logistic regression. In this work, we propose a novel coreset construction via cardinality-constrained bilevel optimization. We show how our framework can efficiently generate coresets for deep neural networks, and demonstrate its empirical benefits in continual learning and in streaming settings.

Citations

• On the limited memory BFGS method for large scale optimization
• Approximation Methods for Bilevel Programming
• DARTS: Differentiable Architecture Search

Cited by

• RS-Prune: Training-Free Data Pruning at High Ratios for Efficient Remote Sensing Diffusion Foundation Models
• Bridging Streaming Continual Learning via In-Context Large Tabular Models
• Online Data Curation for Object Detection via Marginal Contributions to Dataset-level Average Precision
• Explore and Establish Synergistic Effects Between Weight Pruning and Coreset Selection in Neural Network Training
• Compact Memory for Continual Logistic Regression
• Multi-Objective Bilevel Learning
• The brain as a blueprint: a survey of brain-inspired approaches to learning in artificial intelligence
• Dataset Meta-Learning from Kernel Ridge-Regression
• Condensed Composite Memory Continual Learning
• BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretraining
• Optimal Zeroth-Order Bilevel Optimization
• SubZeroCore: A Submodular Approach with Zero Training for Coreset Selection
• First-order Constrained Trilevel Optimization Over Distributed Networks for Robust Coreset Selection
• Carousel Memory: Rethinking the Design of Episodic Memory for Continual Learning
• Bilevel Continual Learning
• The Effectiveness of Memory Replay in Large Scale Continual Learning
• Coresets from Trajectories: Selecting Data via Correlation of Loss Differences
• Beyond Value Functions: Single-Loop Bilevel Optimization under Flatness Conditions
• Dataset Distillation with Infinitely Wide Convolutional Networks
• Recent Advances of Continual Learning in Computer Vision: An Overview
• Information-Theoretic Generalization Bounds of Replay-based Continual Learning
• Online Continual Learning in Image Classification: An Empirical Survey
• The Bayesian Approach to Continual Learning: An Overview
• A resource-efficient method for repeated HPO and NAS problems
• Effective Data Pruning through Score Extrapolation
• Online Coreset Selection for Learning Dynamic Systems
• AdaDeDup: Adaptive Hybrid Data Pruning for Efficient Large-Scale Object Detection Training
• Approximating Language Model Training Data from Weights
• Direct Prediction Set Minimization via Bilevel Conformal Classifier Training
• GORACS: Group-level Optimal Transport-guided Coreset Selection for LLM-based Recommender Systems
• Rethinking Continual Learning with Progressive Neural Collapse
• A Coreset Selection of Coreset Selection Literature: Introduction and Recent Advances
• Accelerating Diagonal Methods for Bilevel Optimization: Unified Convergence via Continuous-Time Dynamics
• FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain
• Perturbed Gradient Descent via Convex Quadratic Approximation for Nonconvex Bilevel Optimization
• Improved Regret Bound and Experience Replay in Regularized Policy Iteration
• CoMBO: Conflict Mitigation via Branched Optimization for Class Incremental Segmentation
• Noise-Tolerant Coreset-Based Class Incremental Continual Learning
• Data Selection for ERMs
• LLM Unlearning Reveals a Stronger-Than-Expected Coreset Effect in Current Benchmarks

Related

• Data Summarization via Bilevel Optimization
• Probabilistic Bilevel Coreset Selection
• Bilevel Continual Learning
• Practical Coreset Constructions for Machine Learning
• PDAC: Efficient Coreset Selection for Continual Learning via Probability Density Awareness