Adaptive Group Policy Optimization: Towards Stable Training and Token-Efficient Reasoning

2025/03/20 by Li Chen, Li, Chen, Kai Yang +2 · 13 citations
Computer Science · #Computation and Language (cs.CL) #Core (optical fiber) #FOS: Computer and information sciences #Group (periodic table) #Inference #Multi-Agent Systems and Negotiation #Security token #Stability (learning theory) #Training (meteorology)

paper · pdf · doi:10.48550/arxiv.2503.15952

openalex publication_date 2025/03/20 · openalex created_date 2025/10/18 · openalex updated_date 2026/08/05

Abstract

Since DeepSeek-R1 popularized, Group Relative Policy Optimization (GRPO) has become the core part of training Reasoning LLMs. However, we find some deficiency that influences RL stability and inference efficiency, like zero-variance in advantage estimation. Thus, we propose Adaptive Group Policy Optimization (AGPO) which uses a simple but effective method, an adaptive loss function, to mitigate training fluctuation and token inefficiency. The experiments demonstrate our method achieves more stable training and superior performance with significantly fewer tokens in reasoning steps.

Cited by

• DeepPrune: Parallel Scaling without Inter-trace Redundancy
• The Markovian Thinker: Architecture-Agnostic Linear Scaling of Reasoning
• GRAD: Generative Retrieval-Aligned Demonstration Sampler for Efficient Few-Shot Reasoning
• STaR-Attack: A Spatio-Temporal and Narrative Reasoning Attack Framework for Unified Multimodal Understanding and Generation Models
• C²GSPG: Confidence-calibrated Group Sequence Policy Gradient towards Self-aware Reasoning
• G²RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidance
• OctoNav: Towards Generalist Embodied Navigation
• Enhancing Large Language Models through Structured Reasoning
• R-KV: Redundancy-aware KV Cache Compression for Reasoning Models
• Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewards
• VeriThinker: Learning to Verify Makes Reasoning Model Efficient
• Activation Control for Efficiently Eliciting Long Chain-of-thought Ability of Language Models
• ThinkFL: Self-Refining Failure Localization for Microservice Systems via Reinforcement Fine-Tuning

Related

• Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPO
• AAPO: Enhancing the Reasoning Capabilities of LLMs with Advantage Momentum
• Mitigating Think-Answer Mismatch in LLM Reasoning Through Noise-Aware Advantage Reweighting
• Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic
• Training-Free Group Relative Policy Optimization