Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset

2024/12/03 by Dan Su, Su, Dan, Kezhi Kong +15 · 53 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Image Processing and 3D Reconstruction #Natural Language Processing Techniques #Semantic Web and Ontologies

paper · pdf · doi:10.48550/arxiv.2412.02595

openalex publication_date 2024/12/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent English Common Crawl datasets like FineWeb-Edu and DCLM achieved significant benchmark gains via aggressive model-based filtering, but at the cost of removing 90% of data. This limits their suitability for long token horizon training, such as 15T tokens for Llama 3.1. In this paper, we show how to achieve better trade-offs between accuracy and data quantity by a combination of classifier ensembling, synthetic data rephrasing, and reduced reliance on heuristic filters. When training 8B parameter models for 1T tokens, using a high-quality subset of our data improves MMLU by 5.6 over DCLM, demonstrating the efficacy of our methods for boosting accuracies over a relatively short token horizon. Furthermore, our full 6.3T token dataset matches DCLM on MMLU, but contains four times more unique real tokens than DCLM. This unlocks state-of-the-art training over a long token horizon: an 8B parameter model trained for 15T tokens, of which 7.2T came from our dataset, is better than the Llama 3.1 8B model: +5 on MMLU, +3.1 on ARC-Challenge, and +0.5 on average across ten diverse tasks. The dataset is available at https://data.commoncrawl.org/contrib/Nemotron/Nemotron-CC/index.html

Cited by

• CuraWeb: Joint Optimization of Quality, Redundancy, and Diversity for Web-Scale Pretraining Data
• Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
• Sigma-MoE-Tiny Technical Report
• FIN-bench-v2: A Unified and Robust Benchmark Suite for Evaluating Finnish Large Language Models
• Scaling Behavior of Discrete Diffusion Language Models
• PCMind-2.1-Kaiyuan-2B Technical Report
• LIME: Making LLM Data More Efficient with Linguistic Metadata Embeddings
• Nanbeige4-3B Technical Report: Exploring the Frontier of Small Language Models
• How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining
• Blu-WERP (Web Extraction and Refinement Pipeline): A Scalable Pipeline for Preprocessing Large Language Model Datasets
• AICC: Parse HTML Finer, Make Models Better -- A 7.3T AI-Ready Corpus Built by a Model-Based HTML Parser
• BhashaKritika: Building Synthetic Pretraining Data at Scale for Indic Languages
• NextFlow: Unified Sequential Modeling Activates Multimodal Understanding and Generation
• Motif 2 12.7B technical report
• Reusing Pre-Training Data at Test Time is a Compute Multiplier
• Diffusion Language Models are Super Data Learners
• HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models
• Detecting Data Contamination in LLMs via In-Context Learning
• Emu3.5: Native Multimodal Models are World Learners
• AgentFrontier: Expanding the Capability Frontier of LLM Agents with ZPD-Guided Data Synthesis
• Relative Scaling Laws for LLMs
• Data-Centric Lessons To Improve Speech-Language Pretraining
• Learning from the Best, Differently: A Diversity-Driven Rethinking on Data Selection
• RePro: Training Language Models to Faithfully Recycle the Web for Pretraining
• KORMo: Korean Open Reasoning Model for Everyone
• Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts
• Where to Begin: Efficient Pretraining via Subnetwork Selection and Distillation
• Mid-Training of Large Language Models: A Survey
• Training Dynamics Impact Post-Training Quantization Robustness
• Mixture of Neuron Experts
• Optimal Scaling Needs Optimal Norm
• Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilization
• MixtureVitae: Open Web-Scale Pretraining Dataset With High Quality Instruction and Reasoning Data Built from Permissive-First Text Sources
• Beyond Repetition: Text Simplification and Curriculum Learning for Data-Constrained Pretraining
• Training Optimal Large Diffusion Language Models
• LOCA: Logical Chain Augmentation for Scientific Corpus Cleaning
• HSS-Synth: Humanities and Social Sciences Data Synthesis for LLMs
• Multi-Head Attention Residuals
• How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data
• Synthetic bootstrapped pretraining
• Open-sci-ref-0.01: open and reproducible reference baselines for language model and dataset comparison
• Intern-S1: A Scientific Multimodal Foundation Model
• OLMoASR: Open Models and Data for Training Robust Speech Recognition Models
• Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Dataset
• NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model
• SupraTok: Cross-Boundary Tokenization for Enhanced Language Model Performance
• BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining
• LinkQA: Synthesizing Diverse QA from Multiple Seeds Strongly Linked by Knowledge Points
• Data Mixing Agent: Learning to Re-weight Domains for Continual Pre-training
• SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment
• Mellum2 Technical Report
• Language Models Improve When Pretraining Data Matches Target Tasks
• PARAM-1 BharatGen 2.9B Model

Related

• BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining
• Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training
• The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
• Does your data spark joy? Performance gains from domain upsampling at the end of training
• GneissWeb: Preparing High Quality Data for LLMs at Scale