DataRater: Meta-Learned Dataset Curation

2025/05/23 by Dan A. Calian, Calian, Dan A., Gregory Farquhar +21 · 3 voices · 4 citations
#stat.ML #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.2505.17895

Abstract

The quality of foundation models depends heavily on their training data. Consequently, great efforts have been put into dataset curation. Yet most approaches rely on manual tuning of coarse-grained mixtures of large buckets of data, or filtering by hand-crafted heuristics. An approach that is ultimately more scalable (let alone more satisfying) is to learn which data is actually valuable for training. This type of meta-learning could allow more sophisticated, fine-grained, and effective curation. Our proposed DataRater is an instance of this idea. It estimates the value of training on any particular data point. This is done by meta-learning using `meta-gradients', with the objective of improving training efficiency on held out data. In extensive experiments across a range of model scales and datasets, we find that using our DataRater to filter data is highly effective, resulting in significantly improved compute efficiency.

Citations

• Optimizing ML Training with Metagradient Descent
• LIMA: Less Is More for Alignment
• LoRA: Low-Rank Adaptation of Large Language Models
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• Attention Is All You Need
• Bag of Tricks for Efficient Text Classification
• Gradient-based Hyperparameter Optimization through Reversible Learning
• Sparse Approximate Solutions to Linear Systems
• MapReduce

Cited by

• Alchemist: Unlocking Efficiency in Text-to-Image Model Training via Meta-Gradient Data Selection
• Uncovering Competency Gaps in Large Language Models and Their Benchmarks
• Holdout-Loss-Based Data Selection for LLM Finetuning via In-Context Learning
• Language Self-Play For Data-Free Training

Discussions

• Excited to share our new paper, "DataRater: Meta-Learned Dataset Curation"! We explore a fundamental question: How can we *automatically* learn which data is most valuable for training foundation mode [bsky, 26 points, 1 comments]
• Could we meta-learn which data to train on? Yes! Does this make LLM training more efficient? Yes! Would you like to know exactly how? arxiv.org/pdf/2505.17895 (come see us at NeurIPS too!) [bsky, 8 points, 0 comments]
• arxiv.org/abs/2505.17895 [bsky, 0 points, 0 comments]

Related

• Dataset2Vec: Learning Dataset Meta-Features
• FREE: Faster and Better Data-Free Meta-Learning
• Towards Meta-learned Algorithm Selection using Implicit Fidelity Information
• Fair Meta-Learning: Learning How to Learn Fairly
• Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples