Zyda-2: a 5 Trillion Token High-Quality Dataset

2024/11/09 by Yury Tokpanov, Tokpanov, Yury, Paolo Glorioso +5 · 5 citations
Computer Science · Medicine · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning in Healthcare #Radiomics and Machine Learning in Medical Imaging

paper · pdf · doi:10.48550/arxiv.2411.06068

openalex publication_date 2024/11/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this technical report, we present Zyda-2: a five trillion token dataset for language model pretraining. Zyda-2 was used to train our Zamba2 series of models which are state-of-the-art for their weight class. We build Zyda-2 by collating high-quality open-source tokens such as FineWeb and DCLM, then distilling them to the highest-quality subset via cross-deduplication and model-based quality filtering. Zyda-2 is released under a permissive open license, and is available at https://huggingface.co/datasets/Zyphra/Zyda-2

Cited by

• Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Models
• Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Dataset
• Language Models Improve When Pretraining Data Matches Target Tasks
• Tokenizer-Agnostic Engram Module
• MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE

Related

• Zyda: A 1.3T Dataset for Open Language Modeling
• Ziya2: Data-centric Learning is All LLMs Need
• YuLan-Mini: An Open Data-efficient Language Model
• The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
• 1.5-Pints Technical Report: Pretraining in Days, Not Months -- Your Language Model Thrives on Quality Data