SAGE: Streaming Agreement-Driven Gradient Sketches for Representative Subset Selection

2025/10/02 by Ashish Kumar Jha, Jha, Ashish, Salman Ahmadi‐Asl +1
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2510.02470

openalex publication_date 2025/10/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Training modern neural networks on large datasets is computationally and energy intensive. We present SAGE, a streaming data-subset selection method that maintains a compact Frequent Directions (FD) sketch of gradient geometry in O(ℓ D) memory and prioritizes examples whose sketched gradients align with a consensus direction. The approach eliminates N × N pairwise similarities and explicit N × ℓ gradient stores, yielding a simple two-pass, GPU-friendly pipeline. Leveraging FD's deterministic approximation guarantees, we analyze how agreement scoring preserves gradient energy within the principal sketched subspace. Across multiple benchmarks, SAGE trains with small kept-rate budgets while retaining competitive accuracy relative to full-data training and recent subset-selection baselines, and reduces end-to-end compute and peak memory. Overall, SAGE offers a practical, constant-memory alternative that complements pruning and model compression for efficient training.

Citations

• Market-Driven Subset Selection for Budgeted Training
• GRAFT: Gradient-Aware Fast MaxVol Technique for Dynamic Data Sampling
• Beyond neural scaling laws: beating power law scaling via data pruning
• Training Compute-Optimal Large Language Models
• Scaling Laws for Neural Language Models
• Gradient Surgery for Multi-Task Learning
• BackPACK: Packing more into backprop
• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
• Data Shapley: Equitable Valuation of Data for Machine Learning
• An Empirical Study of Example Forgetting during Deep Neural Network Learning
• Understanding Black-box Predictions via Influence Functions
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Distilling the Knowledge in a Neural Network
• Frequent Directions : Simple and Deterministic Matrix Sketching
• Lazier Than Lazy Greedy
• Simple and Deterministic Matrix Sketching

Related

• GRAFT: Gradient-Aware Fast MaxVol Technique for Dynamic Data Sampling
• Compressing Gradient Optimizers via Count-Sketches
• Lean and Mean Adaptive Optimization via Subset-Norm and Subspace-Momentum with Convergence Guarantees
• GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training
• MILO: Model-Agnostic Subset Selection Framework for Efficient Model Training and Tuning