Group Sequence Policy Optimization

2025/07/24 by Chujie Zheng, Zheng, Chujie, Shixuan Liu +21 · 4 voices · 194 citations
#cs.LG #cs.AI #cs.CL

paper · pdf · doi:10.48550/arxiv.2507.18071

Abstract

This paper introduces Group Sequence Policy Optimization (GSPO), our stable, efficient, and performant reinforcement learning algorithm for training large language models. Unlike previous algorithms that adopt token-level importance ratios, GSPO defines the importance ratio based on sequence likelihood and performs sequence-level clipping, rewarding, and optimization. We demonstrate that GSPO achieves superior training efficiency and performance compared to the GRPO algorithm, notably stabilizes Mixture-of-Experts (MoE) RL training, and has the potential for simplifying the design of RL infrastructure. These merits of GSPO have contributed to the remarkable improvements in the latest Qwen3 models.

Citations

• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning

Cited by

• Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO
• Reinforcement Learning for Large Language Model Selective Evidence Adoption from Contaminated Retrieval Results
• Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learning
• ArenaRL: Scaling RL for Open-Ended Agents via Tournament-based Relative Ranking
• Molt: A Scalable PyTorch-Native Training Framework for Agentic Reinforcement Learning
• Solar Open 2 Technical Report
• Dr. Zero: Self-Evolving Search Agents without Training Data
• RetroAgent: Harnessing LLMs to Search Over Structured Memory for Agentic Retrosynthesis Planning
• Copy Less, Ground More: Overcoming Repetitive Copying in Long-Context Reasoning via Evidence-Aware Reinforcement Learning
• Implicit Actor Critic Coupling via a Supervised Learning Framework for RLVR
• Mask-Aware Policy Gradients for Diffusion Language Models
• Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning
• Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimization
• VibeThinker-3B: Exploring the Frontier of Verifiable Reasoning in Small Language Models
• Qwen-AgentWorld: Language World Models for General Agents
• PrAg-PO: Prompt Augmented Policy Optimization for Robust and Diverse Mathematical Reasoning
• Let It Flow: Agentic Crafting on Rock and Roll, Building the ROME Model within an Open Agentic Learning Ecosystem
• Agentic Entropy-Balanced Policy Optimization
• Qwen-Music Technical Report
• Reinforcement Learning via Self-Distillation
• GLM-5: from Vibe Coding to Agentic Engineering
• ACRL: Adaptive Control of Training-Inference Discrepancy for Stable Reinforcement Learning
• Self-Boosting Vision-Language Models with Noisy Student On-Policy Self-Distillation
• CAST: Game Solvers as Turn-Level Teachers for LLM Agents
• VlogReward: Learning Multi-Dimensional Evaluation for Vlog Editing
• Multi-Task GRPO: Reliable LLM Reasoning Across Tasks
• A Comedy of Estimators: On KL Regularization in RL Training of LLMs
• Scaling Reinforcement Learning for Content Moderation with Large Language Models
• TableGPT-R1: Advancing Tabular Reasoning Through Reinforcement Learning
• Memory-T1: Reinforcement Learning for Temporal Reasoning in Multi-session Agents
• PuzzleCraft: Exploration-Aware Curriculum Learning for Puzzle-Based RLVR in VLMs
• SAGE: Training Smart Any-Horizon Agents for Long Video Reasoning with Reinforcement Learning
• SocialNav-MoE: A Mixture-of-Experts Vision Language Model for Socially Compliant Navigation with Reinforcement Fine-Tuning
• AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoning
• Toward Ambulatory Vision: Learning Visually-Grounded Active View Selection
• TraPO: A Semi-Supervised Reinforcement Learning Framework for Boosting LLM Reasoning
• LLM-Auction: Generative Auction towards LLM-Native Advertising
• Are We Ready for RL in Text-to-3D Generation? A Progressive Investigation
• d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Models
• Thinking with Images via Self-Calling Agent
• Each Prompt Matters: Scaling Reinforcement Learning Without Wasting Rollouts on Hundred-Billion-Scale MoE
• Native Parallel Reasoner: Reasoning in Parallelism via Self-Distilled Reinforcement Learning
• Qwen3.5-Omni Technical Report
• Efficient Reinforcement Learning with Semantic and Token Entropy for LLM Reasoning
• Thinking with Programming Vision: Towards a Unified View for Thinking with Images
• Semantic Soft Bootstrapping: Long Context Reasoning in LLMs without Reinforcement Learning
• On GRPO Collapse in Search-R1: The Lazy Likelihood-Displacement Death Spiral
• TempR1: Improving Temporal Understanding of MLLMs via Temporal-Aware Multi-Task Reinforcement Learning
• OneThinker: All-in-one Reasoning Model for Image and Video
• MindGPT-4ov: An Enhanced MLLM via a Multi-Stage Post-Training Paradigm
• Rectifying LLM Thought from Lens of Optimization
• ESPO: Entropy Importance Sampling Policy Optimization
• The Image as Its Own Reward: Reinforcement Learning with Adversarial Reward for Image Generation
• Boosting Reasoning in Large Multimodal Models via Activation Replay
• Soft Adaptive Policy Optimization
• Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization
• VADE: Variance-Aware Dynamic Sampling via Online Sample-Level Difficulty Estimation for Multimodal RL
• Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatch
• Thinking-while-Generating: Interleaving Textual Reasoning throughout Visual Generation
• Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learning
• Agent-R1: Training Powerful LLM Agents with End-to-End Reinforcement Learning
• P1: Mastering Physics Olympiads with Reinforcement Learning
• Toward Honest Language Models for Deductive Reasoning
• MSCR: Exploring the Vulnerability of LLMs' Mathematical Reasoning Abilities Using Multi-Source Candidate Replacement
• Numerical Sensitivity and Robustness: Exploring the Flaws of Mathematical Reasoning in Large Language Models
• IterResearch: Rethinking Long-Horizon Agents via Markovian State Reconstruction
• DRIVE: Data Curation Best Practices for Reinforcement Learning with Verifiable Reward in Competitive Code Generation
• FLEX: Continuous Agent Evolution via Forward Learning from Experience
• Explore Data Left Behind in Reinforcement Learning for Reasoning Language Models
• SSPO: Subsentence-level Policy Optimization
• Saliency-R1: Incentivizing Unified Saliency Reasoning Capability in MLLM with Confidence-Guided Reinforcement Learning
• Defeating the Training-Inference Mismatch via FP16
• BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuning
• PORTool: Tool-Use LLM Training with Rewarded Tree
• Grounded in Reality: Learning and Deploying Proactive LLM from Offline Logs
• Reasoning-Aware GRPO using Process Mining
• Lookahead Tree-Based Rollouts for Enhanced Trajectory-Level Exploration in Reinforcement Learning with Verifiable Rewards
• VOLD: Reasoning Transfer from LLMs to Vision-Language Models via On-Policy Distillation
• Advantage Shaping as Surrogate Reward Maximization: Unifying Pass@K Policy Gradients
• Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts
• Agent-GSPO: Communication-Efficient Multi-Agent Systems via Group Sequence Policy Optimization
• Scaling Up Efficient Small Language Models Serving and Deployment for Semantic Job Search
• Sample By Step, Optimize By Chunk: Chunk-Level GRPO For Text-to-Image Generation
• The Virtues of Brevity: Avoid Overthinking in Parallel Test-Time Reasoning
• Rank-GRPO: Training LLM-based Conversational Recommender Systems with Reinforcement Learning
• Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidence
• Code-enabled language models can outperform reasoning models on diverse tasks
• GAPO: Robust Advantage Estimation for Real-World Code LLMs
• Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoning
• A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning
• BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping
• Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views
• InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Training
• Voting with the Graph: Stable RLAIF via Topological Consistency Maximization
• LSPO: Length-aware Dynamic Sampling for Policy Optimization in LLM Reasoning
• Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn LLM Agents
• LaSeR: Reinforcement Learning with Last-Token Self-Rewarding
• Large Reasoning Embedding Models: Towards Next-Generation Dense Retrieval Paradigm
• Qwen3Guard Technical Report
• Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization
• Reducing Belief Deviation in Reinforcement Learning for Active Reasoning
• Reinforced Preference Optimization for Recommendation
• Demystifying Reinforcement Learning in Agentic Reasoning
• QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs
• CoPRS: Learning Positional Prior from Chain-of-Thought for Reasoning Segmentation
• Demystifying Reinforcement Learning for Long-Horizon Tool-Using Agents: A Comprehensive Recipe
• SPG: Sandwiched Policy Gradient for Masked Diffusion Language Models
• DSPO: Stable and Efficient Policy Optimization for Agentic Search and Reasoning
• Pinpointing crucial steps: Attribution-based Credit Assignment for Verifiable Reinforcement Learning
• GCPO: When Contrast Fails, Go Gold
• Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimization
• On the optimization dynamics of RLVR: Gradient gap and step size thresholds
• Beyond Pass@k: Breadth-Depth Metrics for Reasoning Boundaries
• Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimization
• FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts
• Training-Free Group Relative Policy Optimization
• Entropy Regularizing Activation: Boosting Continuous Control, Large Language Models, and Image Classification with Activation as Entropy Constraints
• XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitation
• λ-GRPO: Unifying the GRPO Frameworks with Learnable Token Preferences
• When Importance Sampling Misallocates Credit: Asymmetric Ratios for Outcome-Supervised RL
• Let it Calm: Exploratory Annealed Decoding for Verifiable Reinforcement Learning
• Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoning
• Reinforcement Learning for Clinical Reasoning: Aligning LLMs with ACR Imaging Appropriateness Criteria
• Explore Briefly, Then Decide: Mitigating LLM Overthinking via Cumulative Entropy Regulation
• Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectives
• The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View
• GUI-Spotlight: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Grounding
• TROLL: Trust Regions improve Reinforcement Learning for Large Language Models
• Token Hidden Reward: Steering Exploration-Exploitation in Group Relative Deep Reinforcement Learning
• The Path of Self-Evolving Large Language Models: Achieving Data-Efficient Learning via Intrinsic Feedback
• Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?
• Safety Instincts: LLMs Learn to Trust Their Internal Compass for Self-Defense
• ImageDoctor: Diagnosing Text-to-Image Generation via Grounded Image Reasoning
• It Takes Two: Your GRPO Is Secretly DPO
• RiskPO: Risk-based Policy Optimization via Verifiable Reward for LLM Post-Training
• Thinkquel: A Model Dedicated to Text-to-dbt Using Synthetic Data and a Span-Aware Objective
• Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Models
• Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners
• More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models
• Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse
• CLPO: Curriculum Learning meets Policy Optimization for LLM Reasoning
• Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewards
• Why Tree-Style Branching Matters for Thought Advantage Estimation in GRPO
• Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friends
• Reasoning or Retrieval? A Study of Answer Attribution on Large Reasoning Models
• Reinforcement Learning with Inverse Rewards for World Model Post-training
• Poivre: Self-Refining Visual Pointing with Reinforcement Learning
• Advancing Multi-agent Traffic Simulation via R1-Style Reinforcement Fine-Tuning
• Learning to Reason in Structured In-context Environments with Reinforcement Learning
• Tagging the Thought: Unlocking Personalization Reasoning via Reinforcement Learning
• From Harm to Help: Turning Reasoning In-Context Demos into Assets for Reasoning LMs
• C²GSPG: Confidence-calibrated Group Sequence Policy Gradient towards Self-aware Reasoning
• Variational Reasoning for Language Models
• Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learning
• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Meta-Awareness Enhances Reasoning Models: Self-Alignment Reinforcement Learning
• MultiCrafter: High-Fidelity Multi-Subject Generation via Disentangled Attention and Identity-Aware Preference Alignment
• Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoning
• Learning More with Less: A Dynamic Dual-Level Down-Sampling Framework for Efficient Policy Optimization
• Tree Search for LLM Agent Reinforcement Learning
• Expanding Reasoning Potential in Foundation Model by Learning Diverse Chains of Thought Patterns
• It's Not You, It's Clipping: A Soft Trust-Region via Probability Smoothing for LLM RL
• Future Policy Aware Preference Learning for Mathematical Reasoning
• Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Models
• GEPO: Group Expectation Policy Optimization for Stable Heterogeneous Reinforcement Learning
• Agentic Reinforcement Learning with Implicit Step Rewards
• Failure Makes the Agent Stronger: Enhancing Accuracy through Structured Reflection for Reliable Tool Interactions
• OraPO: Oracle-educated Reinforcement Learning for Data-efficient and Factual Radiology Report Generation
• ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs
• Correlation or Causation: Analyzing the Causal Structures of LLM and LRM Reasoning Process
• Qwen3-Omni Technical Report
• Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
• FlowRL: Matching Reward Distributions for LLM Reasoning
• THOR: Tool-Integrated Hierarchical Optimization via RL for Mathematical Reasoning
• SAIL-VL2 Technical Report
• LLM-I: LLMs are Naturally Interleaved Multimodal Creators
• Single-stream Policy Optimization
• WebResearcher: Unleashing unbounded reasoning capability in Long-Horizon Agents
• Inpainting-Guided Policy Optimization for Diffusion Large Language Models
• Clip Your Sequences Fairly: Enforcing Length Fairness for Sequence-Level RL
• SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agents
• Towards a Unified View of Large Language Model Post-Training
• SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning
• DCPO: Dynamic Clipping Policy Optimization
• Intern-S1: A Scientific Multimodal Foundation Model
• InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
• Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVR
• TaoSR1: The Thinking Model for E-commerce Relevance Search
• On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification
• Prompt Injection Vulnerability of Consensus Generating Applications in Digital Democracy
• GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control
• On the Theory and Practice of GRPO: A Trajectory-Corrected Approach with Fast Convergence
• MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge
• Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

Discussions

• 🚨Great Paper Alert🚨 GSPO (Group Sequence Policy Optimization) tbh it's a bit tough on the math, but it's EXCELLENT at explaining the situation it's an RL algorithm that fixes stability problems with [bsky, 32 points, 1 comments]
• tl;dr: do importance weighting/sampling on a sequence level, not a token level. Makes everything behave much better (see below) and makes more sense from a theoretical perspective, too. Paper: www.arx [bsky, 3 points, 0 comments]
• Group Sequence Policy Optimization [hn, 2 points, 1 comments]
• Group Sequence Policy Optimization 💡 Very interesting paper. arxiv.org/abs/2507.18071 [bsky, 0 points, 0 comments]

Related

• GRPO-RM: Fine-Tuning Representation Models via GRPO-Driven Reinforcement Learning
• SSPO: Subsentence-level Policy Optimization
• Group-Aware Reinforcement Learning for Output Diversity in Large Language Models
• GVPO: Group Variance Policy Optimization for Large Language Model Post-Training
• Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPO