A theoretical basis for model collapse in recursive training

2025/06/11 by Borkar, Vivek Shripad
#68T01 #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Probability (math.PR)

paper · doi:10.48550/arxiv.2506.09401

Abstract

It is known that recursive training from generative models can lead to the so called `collapse' of the simulated probability distribution. This note shows that one in fact gets two different asymptotic behaviours depending on whether an external source, howsoever minor, is also contributing samples.

Citations

• Rate of Model Collapse in Recursive Training
• AI models collapse when trained on recursively generated data
• Heat Death of Generative Models in Closed-Loop Learning
• Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data
• Model Collapse Demystified: The Case of Regression
• Self-Consuming Generative Models Go MAD
• Dynamic / ME-JEPA v2.0.0-rc1: Audited World-Model Runtime and Verified Training-Corpus Artifact

Related

• A Probabilistic Perspective on Model Collapse
• Rate of Model Collapse in Recursive Training
• A Note on Shumailov et al. (2024): `AI Models Collapse When Trained on Recursively Generated Data'
• How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse
• When Models Don't Collapse: On the Consistency of Iterative MLE