Self-Consuming Generative Models Go MAD

2023/07/04 by Sina Alemohammad, Alemohammad, Sina, Josue Casco-Rodriguez +13 · 20 voices · 45 citations
Computer Science · #Generative Adversarial Networks and Image Synthesis

paper · pdf · doi:10.48550/arxiv.2307.01850

Abstract

Model collapse — the degradation of models trained on model-generated data — has been characterized almost entirely at the level of outputs and distributions: perplexity, tail loss, diversity. We report a pre-registered experiment that probes collapse at the level of internal structure, using a measured geometric property of trained attention: the formation of conformal (power-law) attention heads, which prior work in this program showed natural text reliably induces (11–15/48 heads across seeds) while corpora engineered to match natural language's mutual-information statistics (0–5/48) and hierarchical grammar without reference (0/48) do not. We train a fresh Pythia-70m-class model, under an identical pre-registered protocol, on 1.1B tokens generated by the best-performing natural-text model — text carrying the full statistical fingerprint of a model that itself possessed the geometry (15/48 heads), produced at temperature 1.0 with no truncation. The pre-registered formation criterion (≥10/48) fails: the child model forms 7/48 conformal heads, with the deep steep-exponent population largely missing (at this formation scale no model, parent included, reaches the strict SYK-near window — formation count, not matured exponent, is the discriminating observable; see §2). A same-day two-seed replication on the identical corpus (declared before launch) confirms the failure is seed-robust: 7 and 3/48 — a child band of 3–7/48 that does not overlap the natural-text band of 11–15/48. Meanwhile the generated corpus itself has more long-range pairwise mutual information than the natural corpus at essentially every measured distance (post-hoc measurement, pre-registered estimator). The internal geometry collapses in a single generation, at unmodified sampling temperature, while the low-order statistics that existing collapse accounts track actually increase. Together with the earlier rungs of the program's corpus-discrimination ladder, the surviving explanation is that the driver of the conformal phase is language's binding to a persistent external referent — a property that generation by this small model does not preserve. Because the parent corpus (TinyStories) is itself frontier-model-generated, "synthetic" is not the operative variable — preservation of reference is. We propose the attention-geometry census as a cheap, pre-registerable, first-generation structural marker for data-quality and model-collapse studies. Pre-registrations (commits cbd0ab2f, 8454ac0d), code, per-head data, and experiment notes with recorded deviations are in the public repository 3ld0n/attention-geometry.

Cited by

• Prior laundering: learned priors with inherited, undetectable overconfidence
• Content Creation with Spillovers: An Incentive Design Approach
• The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems
• Optimal Self-Distillation for Rectified Flow via Linear Probing
• Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence
• AI as Governance
• Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges
• Position: Don't Just "Fix it in Post": A Science of AI Must Study Training Dynamics
• Market Design for AI: Beyond the Copyright Binary
• Few-Shot Learning of a Graph-Based Neural Network Model Without Backpropagation
• Entropy Collapse: A Universal Failure Mode of Intelligent Systems
• On the Dangers of Bootstrapping Generation for Continual Learning and Beyond
• Chasing Shadows: Pitfalls in LLM Security Research
• Self-Improving VLM Judges Without Human Annotations
• The Necessity of Imperfection:Reversing Model Collapse via Simulating Cognitive Boundedness
• Preventing Model Collapse via Contraction-Conditioned Neural Filters
• Overcoming the Curvature Bottleneck in MeanFlow
• Synthetic Eggs in Many Baskets: The Impact of Synthetic Data Diversity on LLM Fine-Tuning
• Stabilizing Self-Consuming Diffusion Models with Latent Space Filtering
• Forgetting is Everywhere
• SynQuE: Estimating Synthetic Dataset Quality Without Annotations
• Why Less is More (Sometimes): A Theory of Data Curation
• The Moltbook Files: A Harmless Slopocalypse or Humanity's Last Experiment
• Alice v1: Distillation-Enhanced Video Generation Surpassing Closed-Source Models
• Retrieval Collapses When AI Pollutes the Web
• Fine-tuning Flow Matching Generative Models with Intermediate Feedback
• Beyond Real Data: Synthetic Data through the Lens of Regularization
• Evaluating generative AI’s potential to dispel misinformation about wind farms
• Neon: Negative Extrapolation From Self-Training Improves Image Generation
• Deep Generative Continual Learning using Functional LoRA: FunLoRA
• Characterizing Model Behavior Under Synthetic Data Training: An Empirical Study Across Scales and Mixing Ratios
• Learning in an Echo Chamber: Online Learning with Replay Adversary
• Semantic Voting: A Self-Evaluation-Free Approach for Efficient LLM Self-Improvement on Unverifiable Open-ended Tasks
• Preventing Model Collapse Under Overparametrization: Optimal Mixing Ratios for Interpolation Learning and Ridge Regression
• Training-Free Synthetic Data Generation with Dual IP-Adapter Guidance
• Review of Hallucination Understanding in Large Language and Vision Models
• Ghost in the cache: How data decay shapes the unseen landscape of AI memory
• Synthetic data as meaningful data. On Responsibility in data ecosystems
• A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspective
• Benchmarking the Detection of LLMs-Generated Modern Chinese Poetry
• What is a digital author? Generative AI, concealment, and notes for the emergent moment
• Non-Verbal Vocalisations and their Challenges: Emotion, Privacy, Sparseness, and Real Life
• Model Misalignment and Language Change: Traces of AI-Associated Language in Unscripted Spoken English
• Lessons from complex systems science for AI governance
• Flow Matching Policy Gradients

Discussions

• Self-Consuming Generative Models Go MAD [hn, 49 points, 52 comments]
• I was waiting for this point: The risk of generative models creating "Model Autophagy Disorder" (MAD) when training on their own synthetic data. Without fresh real data, future models may see a decl [bsky, 7 points, 2 comments]
• here u go have a primary source arxiv.org/pdf/2307.018... [bsky, 5 points, 1 comments]
• Das Mad AI disease wird schon diskutiert. Allerdings widerspricht dem Ganzen ein wenig der Wissensstand einer frischen Instanz von OpenAI, der derzeit noch mit 09/21 arbeitet. arxiv.org/abs/2307.018 [bsky, 4 points, 1 comments]
• Thinking about how AI models deteriorate when trained on AI generated material while trying to give myself a backrub arxiv.org/abs/2307.01850 [bsky, 3 points, 0 comments]
• arxiv.org/abs/2307.01850 [bsky, 3 points, 5 comments]
• that's not my understanding of this paper, but i believe you when you say it happens a lot and will happen more in the future. AI researchers can be awfully irresponsible lol arxiv.org/abs/2307.01850 [bsky, 2 points, 1 comments]
• Spicy take of the day (since I've been blocked by another user for actually knowing relatively-well what I'm talking about with respect to machine learning): many people are completely misunderstandin [bsky, 2 points, 1 comments]
• their fascinating paper on AI inbreeding (not yet peer reviewed) can be found here: https://arxiv.org/abs/2307.01850 [bsky, 2 points, 0 comments]
• 🧪 Over-reliance on synthetic data can harm AI models, warn Rice and Stanford researchers. Three training types were identified. Too much synthetic data degrades models, but a moderate amount, much li [bsky, 2 points, 0 comments]
• Self-Consuming Generative Models Go Mad [hn, 2 points, 1 comments]
• Ouroboros arxiv.org/abs/2307.01850 www.pcgamer.com/software/ai/... [bsky, 1 points, 0 comments]
• I think this might be what I was hoping to find, too: arxiv.org/abs/2307.01850 [bsky, 1 points, 0 comments]
• "Our primary conclusion across all scenarios is that without enough fresh real data in each generation of an autophagous loop, future generative models are doomed to have their quality (precision) or [bsky, 1 points, 0 comments]
• There’s already a name for it which is a play on the original use of “MAD”… Model Autophagy Disorder (MAD) arxiv.org/abs/2307.01850 [bsky, 0 points, 0 comments]
• Do you know of MAD? Model Autophagy Disorder - AI trained on AI eventually breaks arxiv.org/abs/2307.018... [bsky, 0 points, 0 comments]
• Lastly, we see this as some evidence that, under the right circumstances, LLM-generated data can actually improve them. Also check related work by Alemohammad et al. (arxiv.org/abs/2307.01850) or Sh [bsky, 0 points, 0 comments]
• Source for quote in previous post: Alemohammad, S. et al. Preprint at arXiv doi.org/10.48550/arX... (2023). [bsky, 0 points, 0 comments]
• Seismic advances in GenAI algorithms for imagery, text, & other data types has led to the temptation to use synthetic data to train next-generation models. Repeating this process creates an autophagou [bsky, 0 points, 2 comments]
• all to train models to support projects that rarely do things we've haven't all seen before – completing tasks humans already like doing; synthesizing content we already have a disorganized surplus of [bsky, 0 points, 1 comments]

Related

• When AI Eats Itself: On the Caveats of AI Autophagy
• Learning by Surprise: Adaptive Mitigation of Model Collapse in Large Language Models
• MONAL: Model Autophagy Analysis for Modeling Human-AI Interactions
• Self-Improving Diffusion Models with Synthetic Data
• Neon: Negative Extrapolation From Self-Training Improves Image Generation