Theoretical Proof that Auto-regressive Language Models Collapse when Real-world Data is a Finite Set

2024/12/19 by Lecheng Wang, Wang, Lecheng, Xianjie Shi +13 · 2 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2412.14872

openalex publication_date 2024/12/19 · openalex created_date 2024/12/21 · openalex updated_date 2026/07/28

Abstract

Auto-regressive language models (LMs) have been widely used to generate data in data-scarce domains to train new LMs, compensating for the scarcity of real-world data. Previous work experimentally found that LMs collapse when trained on recursively generated data. This paper presents a theoretical proof: once a corpus (such as a subset of the World Wide Web) begins to incorporate generated data and no new real-world data is added to the corpus, then no matter how small the amount of data each LM generates and contributes to the corpus, LM collapse is inevitable after sufficient time. This finding suggests that attempts to mitigate collapse by limiting the quantity of synthetic data in the corpus are fundamentally insufficient. Instead, avoiding collapse hinges on ensuring the quality of synthetic data.

Cited by

• Epistemic diversity across language models mitigates knowledge collapse
• Deep Graph Learning for Industrial Carbon Emission Analysis and Policy Impact

Related

• How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse
• Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data
• How to Synthesize Text Data without Model Collapse?
• Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World
• Escaping Collapse: The Strength of Weak Data for Large Language Model Training