MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models

2024/06/10 by Zichun Yu, Spandan Das, Yu, Zichun +3 · 34 citations
Computer Science · #Advanced Database Systems and Queries #Artificial intelligence #Computer science #Data Mining Algorithms and Applications #Machine learning #Model selection #Selection (genetic algorithm) #Time Series Analysis and Forecasting

paper · pdf · doi:10.48550/arxiv.2406.06046

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/06/10 · openalex created_date 2024/06/12 · openalex updated_date 2026/07/28

Abstract

Pretraining data selection has the potential to improve language model pretraining efficiency by utilizing higher-quality data from massive web data corpora. Current data selection methods, which rely on either hand-crafted rules or larger reference models, are conducted statically and do not capture the evolving data preferences during pretraining. In this paper, we introduce model-aware data selection with data influence models (MATES), where a data influence model continuously adapts to the evolving data preferences of the pretraining model and then selects the data most effective for the current pretraining progress. Specifically, we collect oracle data influence by locally probing the pretraining model and fine-tune a small data influence model to approximate it accurately. The data influence model then predicts data influence over the whole pretraining corpus and selects the most influential data for the next pretraining stage. Experiments of pretraining 410M and 1B models on the C4 dataset demonstrate that MATES significantly outperforms random data selection on extensive downstream tasks. It doubles the gains achieved by the state-of-the-art data selection approach that leverages larger reference models and reduces the total FLOPs required to reach certain performances by half. Further analyses validate the effectiveness of the locally probed oracle data influence and the approximation with data influence models. Our code is open-sourced at https://github.com/cxcscmu/MATES.

Cited by

• Perplexity-Aware Data Scaling Law: Perplexity Landscapes Predict Performance for Continual Pre-training
• AIR: Post-training Data Selection for Reasoning via Attention Head Influence
• Lightweight Time Series Data Valuation on Time Series Foundation Models via In-Context Finetuning
• Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Scheduler
• RePro: Training Language Models to Faithfully Recycle the Web for Pretraining
• Skill-Targeted Adaptive Training
• BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretraining
• Pushing LLMs to Their Logical Reasoning Bound: The Role of Data Reasoning Intensity
• Winning the Pruning Gamble: A Unified Approach to Joint Sample and Token Pruning for Efficient Supervised Fine-Tuning
• TiKMiX: Take Data Influence into Dynamic Mixture for Language Model Pre-training
• Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretraining
• Influence Guided Context Selection for Effective Retrieval-Augmented Generation
• Toward Efficient Influence Function: Dropout as a Compression Tool
• Exploring Training Data Attribution under Limited Access Constraints
• LAMDAS: LLM as an Implicit Classifier for Domain-specific Data Selection
• A Survey on Open Dataset Search in the LLM Era: Retrospectives and Perspectives
• LLM Data Selection and Utilization via Dynamic Bi-level Optimization
• OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration
• Shaping capabilities with token-level data filtering
• Language Models Improve When Pretraining Data Matches Target Tasks
• DATE-LM: Benchmarking Data Attribution Evaluation for Large Language Models
• RefineX: Learning to Refine Pre-training Data at Scale from Expert-Guided Programs
• Data Efficacy for Language Model Training
• Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning
• Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay
• ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining
• Efficient Data Selection at Scale via Influence Distillation
• Enhancing Training Data Attribution with Representational Optimization
• Small-to-Large Generalization: Data Influences Models Consistently Across Scale
• IDEAL: Data Equilibrium Adaptation for Multi-Capability Language Model Alignment
• Not All Documents Are What You Need for Extracting Instruction Tuning Data
• STRIDE: Training Data Attribution via Sparse Recovery from Subset Perturbations
• QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM Pretraining
• Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models
• Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training
• Do Data Valuations Make Good Data Prices?

Related

• BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretraining
• Group-Level Data Selection for Efficient Pretraining
• Predictive Data Selection: The Data That Predicts Is the Data That Teaches
• Farewell to Aimless Large-scale Pretraining: Influential Subset Selection for Language Model
• Harnessing Diversity for Important Data Selection in Pretraining Large Language Models