DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining
2023/05/17 by Sang Michael Xie, Hieu Pham, Xie, Sang Michael +17 · 58 citations
Computer Science · #Topic Modeling #Natural Language Processing Techniques #Multimodal Machine Learning Applications
paper · pdf · doi:10.48550/arxiv.2305.10429
Abstract
The mixture proportions of pretraining data domains (e.g., Wikipedia, books, web text) greatly affect language model (LM) performance. In this paper, we propose Domain Reweighting with Minimax Optimization (DoReMi), which first trains a small proxy model using group distributionally robust optimization (Group DRO) over domains to produce domain weights (mixture proportions) without knowledge of downstream tasks. We then resample a dataset with these domain weights and train a larger, full-sized model. In our experiments, we use DoReMi on a 280M-parameter proxy model to set the domain weights for training an 8B-parameter model (30x larger) more efficiently. On The Pile, DoReMi improves perplexity across all domains, even when it downweights a domain. DoReMi improves average few-shot downstream accuracy by 6.5% points over a baseline model trained using The Pile's default domain weights and reaches the baseline accuracy with 2.6x fewer training steps. On the GLaM dataset, DoReMi, which has no knowledge of downstream tasks, even matches the performance of using domain weights tuned on downstream tasks.
Cited by
- DecoupleMix: Decoupled Ratio Search and Convex Allocation for Scalable VLM Data Recipes
- Characterizing Narrative Content in Web-scale LLM Pretraining Data
- Bridging Compute- and Data-Optimal Pretraining
- DomainPilot: Domain-Level Loss-Guided Two-Stage Data Mixture Optimization for Efficient Language Model Fine-Tuning
- CuraWeb: Joint Optimization of Quality, Redundancy, and Diversity for Web-Scale Pretraining Data
- Multi-Task GRPO: Reliable LLM Reasoning Across Tasks
- A Data-Centric Approach to Generalizable Speech Deepfake Detection
- E-RayZer: Self-supervised 3D Reconstruction as Spatial Visual Pre-training
- XDoGE: Multilingual Data Reweighting to Enhance Language Inclusivity in LLMs
- ADAPT: Learning Task Mixtures for Budget-Constrained Instruction Tuning
- A Unified Understanding of Offline Data Selection and Online Self-refining Generation for Post-training LLMs
- Sample-Adaptivity Tradeoff in On-Demand Sampling
- Fast Data Attribution for Text-to-Image Models
- Think-at-Hard: Selective Latent Iterations to Improve Reasoning Language Models
- Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
- Sampling and Loss Weights in Multi-Domain Training
- FedRW: Efficient Privacy-Preserving Data Reweighting for Enhancing Federated Learning of Language Models
- Motif 2 12.7B technical report
- LongCat-Flash-Omni Technical Report
- Reasoning Path Divergence: A New Metric and Curation Strategy to Unlock LLM Diverse Thinking
- Revisiting Multilingual Data Mixtures in Language Model Pretraining
- Selective Learning for Deep Time Series Forecasting
- Shift is Good: Mismatched Data Mixing Improves Test Performance
- TritonRL: Training LLMs to Think and Code Triton Without Cheating
- Reporting guideline for chatbot health advice studies: the Chatbot Assessment Reporting Tool (CHART) statement
- ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilinguality
- DaMo: Data Mixing Optimizer in Fine-tuning Multimodal LLMs for Mobile Phone Agents
- ssToken: Self-modulated and Semantic-aware Token Selection for LLM Fine-tuning
- Self-Verifying Reflection Helps Transformers with CoT Reasoning
- HoneyBee: Data Recipes for Vision-Language Reasoners
- LLMs are All You Need? Improving Fuzz Testing for MOJO with Large Language Models
- Skill-Targeted Adaptive Training
- BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretraining
- Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls
- Understanding the Role of Training Data in Test-Time Scaling
- Characterizing Model Behavior Under Synthetic Data Training: An Empirical Study Across Scales and Mixing Ratios
- RL-Guided Data Selection for Language Model Finetuning
- MixtureVitae: Open Web-Scale Pretraining Dataset With High Quality Instruction and Reasoning Data Built from Permissive-First Text Sources
- Data-Efficient Training by Evolved Sampling
- Predicting LLM Reasoning Performance with Small Proxy Model
- Harnessing the Potential of Optimizing Data Mixtures via Bayesian Domain Reweighting
- Teffic-Audio: Tell Fact from Fiction
- Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Training
- The Kinetics of Training: A Driven-Nucleation Rate Law for Emergence, Plasticity Loss, and Circuit Control in Language Models
- Explaining Data Mixing Scaling Laws
- Prior-based Noisy Text Data Filtering: Fast and Strong Alternative For Perplexity
- TiKMiX: Take Data Influence into Dynamic Mixture for Language Model Pre-training
- Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretraining
- Exploring Training Data Attribution under Limited Access Constraints
- DreamPRM-1.5: Unlocking the Potential of Each Instance for Multimodal Process Reward Model Training
- Fin-PRM: A Domain-Specialized Process Reward Model for Financial Reasoning in Large Language Models
- DynamixSFT: Dynamic Mixture Optimization of Instruction Tuning Collections
- CTRL Your Shift: Clustered Transfer Residual Learning for Many Small Datasets
- ADMIRE-BayesOpt: Accelerated Data MIxture RE-weighting for Language Models with Bayesian Optimization
- Large-Small Model Collaborative Framework for Federated Continual Learning
- The Art of Breaking Words: Rethinking Multilingual Tokenizer Design
- LoReUn: Data Itself Implicitly Provides Cues to Improve Machine Unlearning
- Using Scaling Laws for Data Source Utility Estimation in Domain-Specific Pre-Training
Related