Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training

2025/05/28 by Youssef Mroueh, Mroueh, Youssef, Nicolas Dupuis +15 · 8 citations
Psychology · #FOS: Computer and information sciences #Human Resource Development and Performance Evaluation #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · pdf · doi:10.48550/arxiv.2505.22257

openalex publication_date 2025/05/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We revisit Group Relative Policy Optimization (GRPO) in both on-policy and off-policy optimization regimes. Our motivation comes from recent work on off-policy Proximal Policy Optimization (PPO), which improves training stability, sampling efficiency, and memory usage. In addition, a recent analysis of GRPO suggests that estimating the advantage function with off-policy samples could be beneficial. Building on these observations, we adapt GRPO to the off-policy setting. We show that both on-policy and off-policy GRPO objectives yield an improvement in the reward. This result motivates the use of clipped surrogate objectives in the off-policy version of GRPO. We then compare the empirical performance of reinforcement learning with verifiable rewards in post-training using both GRPO variants. Our results show that off-policy GRPO either significantly outperforms or performs on par with its on-policy counterpart.

Cited by

• Multi-Reward GRPO Fine-Tuning for De-biasing Large Language Models: A Study Based on Chinese-Context Discrimination Data
• POPI: Personalizing LLMs via Optimized Natural Language Preference Inference
• Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradients
• Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning
• PARL-MT: Learning to Call Functions in Multi-Turn Conversation with Progress Awareness
• FastGRPO: Accelerating Policy Optimization via Concurrency-aware Speculative Decoding and Online Draft Learning
• OraPO: Oracle-educated Reinforcement Learning for Data-efficient and Factual Radiology Report Generation
• AVATAR: Reinforcement Learning to See, Hear, and Reason Over Video

Related

• Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments
• Hybrid Group Relative Policy Optimization: A Multi-Sample Approach to Enhancing Policy Optimization
• RRPO: Reference-Relative Policy Optimization with Stratified Conditional Rollouts
• Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friends
• Transductive Off-policy Proximal Policy Optimization