Group Policy Gradient

2025/10/04 by Junhua Chen, Chen, Junhua, Zixi Zhang +4
Computer Science · Decision Sciences · #Adaptive Dynamic Programming Control #Advanced Bandit Algorithms Research #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2510.03679

openalex publication_date 2025/10/04 · openalex created_date 2025/10/09 · openalex updated_date 2026/07/28

Abstract

We introduce Group Policy Gradient (GPG), a family of critic-free policy-gradient estimators for general MDPs. Inspired by the success of GRPO's approach in Reinforcement Learning from Human Feedback (RLHF), GPG replaces a learned value function with a group-based Monte Carlo advantage estimator, removing the memory, compute, and hyperparameter costs of training a critic while preserving PPO's clipped-objective structure. We prove the consistency of the GPG estimator, analyze the bias-variance tradeoffs, and demonstrate empirically that GPG matches or outperforms PPO on standard benchmarks. GPG makes better use of parallel simulations, which, together with its critic-free design, results in more efficient use of computational resources than PPO.

Citations

• Group-in-Group Policy Optimization for LLM Agent Training
• Hybrid Group Relative Policy Optimization: A Multi-Sample Approach to Enhancing Policy Optimization
• Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Review
• A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Training language models to follow instructions with human feedback
• Trajectory balance: Improved credit assignment in GFlowNets
• VarGrad: A Low-Variance Gradient Estimator for Variational Inference
• Proximal Policy Optimization Algorithms
• High-Dimensional Continuous Control Using Generalized Advantage\n Estimation
• REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization
• Reinforcement Learning: An Introduction

Related

• Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments
• GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning
• Phasic Policy Gradient
• PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learning
• GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control