Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training

2025/05/28 by Youssef Mroueh, Mroueh, Youssef, Nicolas Dupuis +15 · 11 citations
Psychology · #FOS: Computer and information sciences #Function (biology) #Group (periodic table) #Human Resource Development and Performance Evaluation #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement learning #Sample (material) #Sampling (signal processing) #Training (meteorology) #Verifiable secret sharing

paper · pdf · doi:10.48550/arxiv.2505.22257

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/05/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

We revisit Group Relative Policy Optimization (GRPO) in both on-policy and off-policy optimization regimes. Our motivation comes from recent work on off-policy Proximal Policy Optimization (PPO), which improves training stability, sampling efficiency, and memory usage. In addition, a recent analysis of GRPO suggests that estimating the advantage function with off-policy samples could be beneficial. Building on these observations, we adapt GRPO to the off-policy setting. We show that both on-policy and off-policy GRPO objectives yield an improvement in the reward. This result motivates the use of clipped surrogate objectives in the off-policy version of GRPO. We then compare the empirical performance of reinforcement learning with verifiable rewards in post-training using both GRPO variants. Our results show that off-policy GRPO either significantly outperforms or performs on par with its on-policy counterpart.

Citations

• Understanding R1-Zero-Like Training: A Critical Perspective
• Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification
• What is the Alignment Objective of GRPO?
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training
• Transductive Off-policy Proximal Policy Optimization
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• Training language models to follow instructions with human feedback
• Generalized Proximal Policy Optimization with Sample Reuse
• Training Verifiers to Solve Math Word Problems
• Learning to summarize from human feedback
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• P3O: Policy-on Policy-off Policy Optimization
• Proximal Policy Optimization Algorithms
• Deep reinforcement learning from human preferences
• Sample Efficient Actor-Critic with Experience Replay
• Trust Region Policy Optimization

Cited by

• Multi-Reward GRPO Fine-Tuning for De-biasing Large Language Models: A Study Based on Chinese-Context Discrimination Data
• POPI: Personalizing LLMs via Optimized Natural Language Preference Inference
• Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradients
• Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning
• PARL-MT: Learning to Call Functions in Multi-Turn Conversation with Progress Awareness
• FastGRPO: Accelerating Policy Optimization via Concurrency-aware Speculative Decoding and Online Draft Learning
• OraPO: Oracle-educated Reinforcement Learning for Data-efficient and Factual Radiology Report Generation
• AVATAR: Reinforcement Learning to See, Hear, and Reason Over Video
• KnowRL: Exploring Knowledgeable Reinforcement Learning for Factuality
• Gradient Extrapolation-Based Policy Optimization
• Large models for machinery fault diagnosis: Current advances and future directions

Related

• Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments
• Hybrid Group Relative Policy Optimization: A Multi-Sample Approach to Enhancing Policy Optimization
• RRPO: Reference-Relative Policy Optimization with Stratified Conditional Rollouts
• Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friends
• Transductive Off-policy Proximal Policy Optimization