GPG: Generalized Policy Gradient Theorem for Transformer-based Policies

2025/12/11 by Hangyu Mao, Mao, Hangyu, Zhicheng Dou +2
Computer Science · #Machine Learning and Data Classification #Reinforcement Learning in Robotics #Machine Learning and Algorithms

paper · pdf · doi:10.48550/arxiv.2512.10365

Abstract

We present the Generalized Policy Gradient (GPG) Theorem, specifically designed for Transformer-based policies. Notably, we demonstrate that both standard Policy Gradient Theorem and GRPO emerge as special cases within our GPG framework. Furthermore, we explore its practical applications in training Large Language Models (LLMs), offering new insights into efficient policy optimization.

Citations

• Agentic Entropy-Balanced Policy Optimization
• Agentic Reinforced Policy Optimization
• Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learning
• DualRAG: A Dual-Process Approach to Integrate Reasoning and Retrieval for Multi-Hop Question Answering
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• General Table Question Answering via Answer-Formula Joint Generation
• WebWalker: Benchmarking LLMs in Web Traversal
• Search-o1: Agentic Search-Enhanced Large Reasoning Models
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training
• DMQR-RAG: Diverse Multi-Query Rewriting for RAG
• The Llama 3 Herd of Models
• SQL-to-Schema Enhances Schema Linking in Text-to-SQL
• PET-SQL: A Prompt-Enhanced Two-Round Refinement of Text-to-SQL with Cross-consistency
• SheetAgent: Towards A Generalist Agent for Spreadsheet Reasoning and Manipulation via Large Language Models
• SQLBench: A Comprehensive Evaluation for Text-to-SQL Capabilities of Large Language Models
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• A Survey of Reinforcement Learning from Human Feedback
• Controlling Large Language Model-based Agents for Large-Scale Decision-Making: An Actor-Critic Approach
• Let's Verify Step by Step
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Measuring and Narrowing the Compositionality Gap in Language Models
• MuSiQue: Multihop Questions via Single-hop Question Composition
• Measuring Mathematical Problem Solving With the MATH Dataset
• Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
• Proximal Policy Optimization Algorithms
• Attention Is All You Need
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Trust Region Policy Optimization
• REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization
• TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usage

Related

• GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control
• An operator view of policy gradient methods
• On the Theory and Practice of GRPO: A Trajectory-Corrected Approach with Fast Convergence
• Group Policy Gradient
• GVPO: Group Variance Policy Optimization for Large Language Model Post-Training