DiverseGRPO: Mitigating Mode Collapse in Image Generation via Diversity-Aware GRPO

2025/12/25 by H. M. Liu, Huijuan Huang, Liu, Henglin +9
Computer Science · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Image Enhancement Techniques

paper · doi:10.48550/arxiv.2512.21514

openalex publication_date 2025/12/25 · openalex created_date 2025/12/30 · openalex updated_date 2026/07/28

Abstract

Reinforcement learning (RL), particularly GRPO, improves image generation quality significantly by comparing the relative performance of images generated within the same group. However, in the later stages of training, the model tends to produce homogenized outputs, lacking creativity and visual diversity, which restricts its application scenarios. This issue can be analyzed from both reward modeling and generation dynamics perspectives. First, traditional GRPO relies on single-sample quality as the reward signal, driving the model to converge toward a few high-reward generation modes while neglecting distribution-level diversity. Second, conventional GRPO regularization neglects the dominant role of early-stage denoising in preserving diversity, causing a misaligned regularization budget that limits the achievable quality--diversity trade-off. Motivated by these insights, we revisit the diversity degradation problem from both reward modeling and generation dynamics. At the reward level, we propose a distributional creativity bonus based on semantic grouping. Specifically, we construct a distribution-level representation via spectral clustering over samples generated from the same caption, and adaptively allocate exploratory rewards according to group sizes to encourage the discovery of novel visual modes. At the generation level, we introduce a structure-aware regularization, which enforces stronger early-stage constraints to preserve diversity without compromising reward optimization efficiency. Experiments demonstrate that our method achieves a 13%--18% improvement in semantic diversity under matched quality scores, establishing a new Pareto frontier between image quality and diversity for GRPO-based image generation.

Citations

• BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Models
• Coefficients-Preserving Sampling for Reinforcement Learning with Flow Matching
• TempFlow-GRPO: When Timing Matters for GRPO in Flow Models
• HPSv3: Towards Wide-Spectrum Human Preference Score
• MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE
• The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
• Skywork Open Reasoner 1 Technical Report
• Creative Preference Optimization
• DanceGRPO: Unleashing GRPO on Visual Generation
• Flow-GRPO: Training Flow Matching Models via Online RL
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• Unified Reward Model for Multimodal Understanding and Generation
• Diverse Preference Optimization
• Image Generation Diversity Issues and How to Tame Them
• Consistency-diversity-realism Pareto fronts of conditional image generative models
• Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
• Diffusion Model Alignment Using Direct Preference Optimization
• Quality Diversity through Human Feedback: Towards Open-Ended Diversity-Driven Optimization
• Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
• Training Diffusion Models with Reinforcement Learning
• Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation
• Diffusion Models Generate Images Like Painters: an Analytical Theory of Outline First, Details Later
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• Denoising Diffusion Probabilistic Models
• Proximal Policy Optimization Algorithms

Related

• From Competition to Synergy: Unlocking Reinforcement Learning for Subject-Driven Image Generation
• STAGE: Stable and Generalizable GRPO for Autoregressive Image Generation
• DanceGRPO: Unleashing GRPO on Visual Generation
• AR-GRPO: Training Autoregressive Image Generation Models via Reinforcement Learning
• Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learning