Expand and Prune: Maximizing Trajectory Diversity for Effective GRPO in Generative Models

2025/12/17 by Ge, Shiran, Huang, Chenyi, Ai, Yuang +3
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2512.15347

Abstract

Group Relative Policy Optimization (GRPO) is a powerful technique for aligning generative models, but its effectiveness is bottlenecked by the conflict between large group sizes and prohibitive computational costs. In this work, we investigate the trade-off through empirical studies, yielding two key observations. First, we discover the reward clustering phenomenon in which many trajectories collapse toward the group-mean reward, offering limited optimization value. Second, we design a heuristic strategy named Optimal Variance Filtering (OVF), and verify that a high-variance subset of trajectories, selected by OVF can outperform the larger, unfiltered group. However, this static, post-sampling OVF approach still necessitates critical computational overhead, as it performs unnecessary sampling for trajectories that are ultimately discarded. To resolve this, we propose Pro-GRPO (Proactive GRPO), a novel dynamic framework that integrates latent feature-based trajectory pruning into the sampling process. Through the early termination of reward-clustered trajectories, Pro-GRPO reduces computational overhead. Leveraging its efficiency, Pro-GRPO employs an "Expand-and-Prune" strategy. This strategy first expands the size of initial sampling group to maximize trajectory diversity, then it applies multi-step OVF to the latents, avoiding prohibitive computational costs. Extensive experiments on both diffusion-based and flow-based models demonstrate the generality and effectiveness of our Pro-GRPO framework.

Citations

• STAGE: Stable and Generalizable GRPO for Autoregressive Image Generation
• Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learning
• AR-GRPO: Training Autoregressive Image Generation Models via Reinforcement Learning
• TempFlow-GRPO: When Timing Matters for GRPO in Flow Models
• MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE
• Inversion-DPO: Precise and Efficient Post-Training for Diffusion Models
• ReasonGen-R1: CoT for Autoregressive Image generation models through SFT and RL
• Fine-Tuning Next-Scale Visual Autoregressive Models with Group Relative Policy Optimization
• RePrompt: Reasoning-Augmented Reprompting for Text-to-Image Generation via Reinforcement Learning
• GoT-R1: Unleashing Reasoning Capability of MLLM for Visual Generation with Reinforcement Learning
• DanceGRPO: Unleashing GRPO on Visual Generation
• Flow-GRPO: Training Flow Matching Models via Online RL
• SimpleAR: Pushing the Frontier of Autoregressive Visual Generation through Pretraining, SFT, and RL
• Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards
• Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Step
• Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control
• Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Diffusion Model Alignment Using Direct Preference Optimization
• GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment
• Reinforcement Learning for Generative AI: A Survey
• SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
• Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Training Diffusion Models with Reinforcement Learning
• Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation
• Stochastic Interpolants: A Unifying Framework for Flows and Diffusions
• Flow Matching for Generative Modeling
• Building Normalizing Flows with Stochastic Interpolants
• Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
• DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
• Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
• High-Resolution Image Synthesis with Latent Diffusion Models
• Learning Transferable Visual Models From Natural Language Supervision
• Score-Based Generative Modeling through Stochastic Differential Equations
• Denoising Diffusion Probabilistic Models
• Generative Modeling by Estimating Gradients of the Data Distribution
• Proximal Policy Optimization Algorithms
• Human Preference Score: Better Aligning Text-to-Image Models with Human Preference

Cited by

• Scone: Bridging Composition and Distinction in Subject-Driven Image Generation via Unified Understanding-Generation Modeling

Related

• Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO
• GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control
• BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Models
• On the Theory and Practice of GRPO: A Trajectory-Corrected Approach with Fast Convergence
• Neighbor GRPO: Contrastive ODE Policy Optimization Aligns Flow Models