Deduplicating Training Data Makes Language Models Better
2021/07/14 by Katherine Lee, Daphne Ippolito, Lee, Katherine +11 · 58 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2107.06499
openalex publication_date 2021/07/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Abstract
We find that existing language modeling datasets contain many near-duplicate examples and long repetitive substrings. As a result, over 1% of the unprompted output of language models trained on these datasets is copied verbatim from the training data. We develop two tools that allow us to deduplicate training datasets -- for example removing from C4 a single 61 word English sentence that is repeated over 60,000 times. Deduplication allows us to train models that emit memorized text ten times less frequently and require fewer train steps to achieve the same or better accuracy. We can also reduce train-test overlap, which affects over 4% of the validation set of standard datasets, thus allowing for more accurate evaluation. We release code for reproducing our work and performing dataset deduplication at https://github.com/google-research/deduplicate-text-datasets.
Cited by
- Cross-Attention Calibrated Deduplication for Retrieval-Augmented Generation System
- From Data to Device: ELMOD An Efficient German-First 2.7B Language Model for Mobile Inference
- Beyond Shapley: An Influence-Based Data Auditing Pipeline for LLM Alignment and Evaluation
- CuraWeb: Joint Optimization of Quality, Redundancy, and Diversity for Web-Scale Pretraining Data
- AraMix: Recycling, Refiltering, and Deduplicating to Deliver the Largest Arabic Pretraining Corpus
- Large language models are not about natural language
- Black-Box Behavioral Distillation Breaks Safety Alignment in Medical LLMs
- Persian-Phi: Efficient Cross-Lingual Adaptation of Compact LLMs via Curriculum Learning
- PrivCode: When Code Generation Meets Differential Privacy
- Randomized Masked Finetuning: An Efficient Way to Mitigate Memorization of PIIs in LLMs
- Measuring the Impact of Lexical Training Data Coverage on Hallucination Detection in Large Language Models
- Blu-WERP (Web Extraction and Refinement Pipeline): A Scalable Pipeline for Preprocessing Large Language Model Datasets
- Hierarchical Memorization in Large Language Models: Evidence from Citation Generation
- FedRW: Efficient Privacy-Preserving Data Reweighting for Enhancing Federated Learning of Language Models
- HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models
- Hallucinations in Bibliographic Recommendation: Citation Frequency as a Proxy for Training Data Redundancy
- Beyond Line-Level Filtering for the Pretraining Corpora of LLMs
- WAON: Large-Scale and High-Quality Japanese Image-Text Pair Dataset for Vision-Language Models
- Building Trust in Clinical LLMs: Bias Analysis and Dataset Transparency
- Learning from the Best, Differently: A Diversity-Driven Rethinking on Data Selection
- ssToken: Self-modulated and Semantic-aware Token Selection for LLM Fine-tuning
- Leave It to the Experts: Detecting Knowledge Distillation via MoE Expert Signatures
- Utility-Diversity Aware Online Batch Selection for LLM Supervised Fine-tuning
- Sparse Subnetwork Enhancement for Underrepresented Languages in Large Language Models
- An Investigation of Memorization Risk in Healthcare Foundation Models
- Enabling Doctor-Centric Medical AI with LLMs through Workflow-Aligned Tasks and Benchmarks
- Distilling Lightweight Language Models for C/C++ Vulnerabilities
- Empirical Comparison of Membership Inference Attacks in Deep Transfer Learning
- BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretraining
- Large Language Models Hallucination: A Comprehensive Survey
- IndiCASA: A Dataset and Bias Evaluation Framework in LLMs Using Contrastive Embedding Similarity in the Indian Context
- Agent Fine-tuning through Distillation for Domain-specific LLMs in Microdomains
- Scaling Laws Revisited: Modeling the Role of Data Quality in Language Model Pretraining
- RL-Guided Data Selection for Language Model Finetuning
- MixtureVitae: Open Web-Scale Pretraining Dataset With High Quality Instruction and Reasoning Data Built from Permissive-First Text Sources
- Mitigating Hallucination in Multimodal LLMs with Layer Contrastive Decoding
- DynaMIC: Dynamic Multimodal In-Context Learning Enabled Embodied Robot Counterfactual Resistance Ability
- Towards Multimodal Active Learning: Efficient Learning with Limited Paired Data
- On Code-Induced Reasoning in LLMs
- Are Language Models Models?
- Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Training
- Memory in Large Language Models: Mechanisms, Evaluation and Evolution
- Model selection meets clinical semantics: Optimizing ICD-10-CM prediction via LLM-as-Judge evaluation, redundancy-aware sampling, and section-aware fine-tuning
- On the de-duplication of the Lakh MIDI dataset
- Enterprise AI Must Enforce Participant-Aware Access Control
- Scrub It Out! Erasing Sensitive Memorization in Code Language Models via Machine Unlearning
- Building High-Quality Datasets for Portuguese LLMs: From Common Crawl Snapshots to Industrial-Grade Corpora
- Towards EnergyGPT: A Large Language Model Specialized for the Energy Sector
- Mitigating Multimodal Hallucinations via Gradient-based Self-Reflection
- Near-Duplicate Text Alignment under Weighted Jaccard Similarity
- Guidelines for Empirical Studies in Software Engineering involving Large Language Models
- A Study of Privacy-preserving Language Modeling Approaches
- Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training
- Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Dataset
- Learning with Confidence
- Assessing and Mitigating Data Memorization Risks in Fine-Tuned Large Language Models
- Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lens
- Provably Secure Retrieval-Augmented Generation
Related