Convergence and Stability Analysis of Self-Consuming Generative Models with Heterogeneous Human Curation

2025/11/12 by Zhao, Hongru, Fu, Jinwen, Pham, Tuan
#37N40 #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.2511.09002

Abstract

Self-consuming generative models have received significant attention over the last few years. In this paper, we study a self-consuming generative model with heterogeneous preferences that is a generalization of the model in Ferbach et al. (2024). The model is retrained round by round using real data and its previous-round synthetic outputs. The asymptotic behavior of the retraining dynamics is investigated across four regimes using different techniques including the nonlinear Perron--Frobenius theory. Our analyses improve upon that of Ferbach et al. (2024) and provide convergence results in settings where the well-known Banach contraction mapping arguments do not apply. Stability and non-stability results regarding the retraining dynamics are also given.

Citations

• The Road to Artificial SuperIntelligence: A Comprehensive Survey of Superalignment
• Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferences
• Online DPO: Online Direct Preference Optimization with Fast-Slow Chasing
• SimPO: Simple Preference Optimization with a Reference-Free Reward
• Iterative Reasoning Preference Optimization
• RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation
• ORPO: Monolithic Preference Optimization without Reference Model
• Diffusion Models as Constrained Samplers for Optimization with Unknown Constraints
• A Survey of Reinforcement Learning from Human Feedback
• Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
• Nash Learning from Human Feedback
• A density estimation perspective on learning from pairwise human preferences
• Into the LAIONs Den: Investigating Hate in Multimodal Datasets
• A General Theoretical Paradigm to Understand Learning from Human Preferences
• Understanding the Effects of RLHF on LLM Generalisation and Diversity
• On the Stability of Iterative Retraining of Generative Models on their own Data
• Large Language Model Alignment: A Survey
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• OpenAssistant Conversations -- Democratizing Large Language Model Alignment
• Combining Generative Artificial Intelligence (AI) and the Internet: Heading towards Evolution or Degradation?
• Better Diffusion Models Further Improve Adversarial Training
• Benchmarks and Algorithms for Offline Preference-Based Reward Learning
• Self-Instruct: Aligning Language Models with Self-Generated Instructions
• Will Large-scale Generative Models Corrupt Future Datasets?
• Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models
• LAION-5B: An open large-scale dataset for training next generation image-text models
• Deep Generative Modeling on Limited Data with Regularization by Nontransferable Pre-trained Models
• DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• Training language models to follow instructions with human feedback
• Red Teaming Language Models with Language Models
• WebGPT: Browser-assisted question-answering with human feedback
• Multimodal datasets: misogyny, pornography, and malignant stereotypes
• PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training
• Learning to summarize from human feedback
• Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
• Deep reinforcement learning from human preferences
• Professor Forcing: A New Algorithm for Training Recurrent Networks
• RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS

Related

• Towards Theoretical Understandings of Self-Consuming Generative Models
• Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferences
• A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loops
• Heat Death of Generative Models in Closed-Loop Learning
• Convergence Dynamics and Stabilization Strategies of Co-Evolving Generative Models