Reinforcement Learning Meets Masked Generative Models: Mask-GRPO for Text-to-Image Generation

2025/10/15 by Yifu Luo, Xinhao Hu, Luo, Yifu +14 · 2 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2510.13418

openalex publication_date 2025/10/15 · openalex created_date 2025/10/17 · openalex updated_date 2026/07/28

Abstract

Reinforcement learning (RL) has garnered increasing attention in text-to-image (T2I) generation. However, most existing RL approaches are tailored to either diffusion models or autoregressive models, overlooking an important alternative: masked generative models. In this work, we propose Mask-GRPO, the first method to incorporate Group Relative Policy Optimization (GRPO)-based RL into this overlooked paradigm. Our core insight is to redefine the transition probability, which is different from current approaches, and formulate the unmasking process as a multi-step decision-making problem. To further enhance our method, we explore several useful strategies, including removing the KL constraint, applying the reduction strategy, and filtering out low-quality samples. Using Mask-GRPO, we improve a base model, Show-o, with substantial improvements on standard T2I benchmarks and preference alignment, outperforming existing state-of-the-art approaches. The code is available on https://github.com/xingzhejun/Mask-GRPO

Citations

• Wavelet Fourier Diffuser: Frequency-Aware Diffusion Model for Reinforcement Learning
• Lifelong Safety Alignment for Language Models
• Reinforcement Fine-Tuning Powers Reasoning Capability of Multimodal Large Language Models
• DanceGRPO: Unleashing GRPO on Visual Generation
• Flow-GRPO: Training Flow Matching Models via Online RL
• SimpleAR: Pushing the Frontier of Autoregressive Visual Generation through Pretraining, SFT, and RL
• VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning
• GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image Generation
• Understanding R1-Zero-Like Training: A Critical Perspective
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization
• Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models
• Visual-RFT: Visual Reinforcement Fine-Tuning
• Safety Reasoning with Guidelines
• PerPO: Perceptual Preference Optimization via Discriminative Rewarding
• Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Step
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Democratizing Text-to-Image Masked Generative Models with Compact Text-Aware One-Dimensional Tokens
• Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations
• Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis
• GPT-4o System Card
• Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation
• Unlocking the Capabilities of Masked Generative Models for Image Synthesis via Self-Guidance
• Meissonic: Revitalizing Masked Generative Transformers for Efficient High-Resolution Text-to-Image Synthesis
• Generalizing Alignment Paradigm of Text-to-Image Generation with Preferences through f-divergence Minimization
• Show-o: One Single Transformer to Unify Multimodal Understanding and Generation
• Probing the Safety Response Boundary of Large Language Models via Unsafe Decoding Path Generation
• LLaVA-OneVision: Easy Visual Task Transfer
• Autoregressive Image Generation without Vector Quantization
• mDPO: Conditional Preference Optimization for Multimodal Large Language Models
• Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation
• Chameleon: Mixed-Modal Early-Fusion Foundation Models
• SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation
• Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction
• Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
• Step-On-Feet Tuning: Scaling Self-Alignment of LLMs via Bootstrapping
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment
• DreamLLM: Synergistic Multimodal Comprehension and Creation
• SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Training Diffusion Models with Reinforcement Learning
• ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation
• GPT-4 Technical Report
• Scaling up GANs for Text-to-Image Synthesis
• LLaMA: Open and Efficient Foundation Language Models
• Muse: Text-To-Image Generation via Masked Generative Transformers
• Scalable Diffusion Models with Transformers
• Rethinking the Objectives of Vector-Quantized Tokenizers for Image Synthesis
• Unified Discrete Diffusion for Simultaneous Vision-Language Generation
• MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis
• LAION-5B: An open large-scale dataset for training next generation image-text models
• Flow Matching for Generative Modeling
• Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
• Scaling Autoregressive Models for Content-Rich Text-to-Image Generation
• Hierarchical Text-Conditional Image Generation with CLIP Latents
• Training language models to follow instructions with human feedback
• MaskGIT: Masked Generative Image Transformer
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• Diffusion Models Beat GANs on Image Synthesis
• Learning Transferable Visual Models From Natural Language Supervision
• Zero-Shot Text-to-Image Generation
• Taming Transformers for High-Resolution Image Synthesis
• Denoising Diffusion Implicit Models
• Denoising Diffusion Probabilistic Models
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• Image Transformer
• Proximal Policy Optimization Algorithms
• Attention Is All You Need
• Microsoft COCO: Common Objects in Context
• OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cycles
• OpenAI o1 System Card
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• Co-GRPO: Co-Optimized Group Relative Policy Optimization for Masked Diffusion Model
• Sample By Step, Optimize By Chunk: Chunk-Level GRPO For Text-to-Image Generation

Related

• MaskFocus: Focusing Policy Optimization on Critical Steps for Masked Image Generation
• AR-GRPO: Training Autoregressive Image Generation Models via Reinforcement Learning
• Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learning
• Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPO
• STAGE: Stable and Generalizable GRPO for Autoregressive Image Generation