Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoning

2025/10/02 by Johan Obando-Ceron, Liu, Jiashun, Obando-Ceron, Johan +16 · 2 citations
Decision Sciences · #Artificial Intelligence (cs.AI) #Auction Theory and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2510.01656

openalex publication_date 2025/10/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Most recent RL for LLMs (RL4LLM) methods avoid explicit critics, replacing them with average advantage baselines. This shift is largely pragmatic: conventional value functions are computationally expensive to train at LLM scale and often fail under sparse rewards and long reasoning horizons. We revisit this bottleneck from an architectural perspective and introduce Asymmetric Proximal Policy Optimization (AsyPPO), a simple and scalable framework that restores the critics role while remaining efficient in large-model settings. AsyPPO employs a set of lightweight mini-critics, each trained on disjoint prompt shards. This design encourages diversity while preserving calibration, reducing value-estimation bias. Beyond robust estimation, AsyPPO leverages inter-critic uncertainty to refine the policy update: (i) masking advantages in states where critics agree and gradients add little learning signal, and (ii) filtering high-divergence states from entropy regularization, suppressing spurious exploration. After training on open-source data with only 5,000 samples, AsyPPO consistently improves learning stability and performance across multiple benchmarks over strong baselines, such as GRPO, achieving performance gains of more than six percent on Qwen3-4b-Base and about three percent on Qwen3-8b-Base and Qwen3-14b-Base over classic PPO, without additional tricks. These results highlight the importance of architectural innovations for scalable, efficient algorithms.

Citations

• A Survey of Reinforcement Learning for Large Reasoning Models
• SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning
• Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning
• VRPO: Rethinking Value Modeling for Robust RL Training under Noisy Supervision
• Geometric-Mean Policy Optimization
• Group Sequence Policy Optimization
• Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning
• Truncated Proximal Policy Optimization
• MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention
• The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning
• Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library
• TL;DR: Too Long, Do Re-weighting for Efficient LLM Reasoning Compression
• Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
• Optimistic critics can empower small actors
• AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning
• Discriminative Policy Optimization for Token-Level Reward Models
• The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
• Improving Value Estimation Critically Enhances Vanilla Policy Gradient
• Qwen3 Technical Report
• DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning
• VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks
• Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model
• Understanding R1-Zero-Like Training: A Critical Perspective
• SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret
• Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs
• Process Reinforcement through Implicit Rewards
• Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage Policy Optimization
• Free Process Rewards without Process Labels
• GPT-4o System Card
• SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning
• Neuroplastic Expansion in Deep Reinforcement Learning
• The Llama 3 Herd of Models
• Controlling Large Language Model Agents with Entropic Activation Steering
• OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Let's Verify Step by Step
• Bigger, Better, Faster: Human-level Atari with human-level efficiency
• Solving Quantitative Reasoning Problems with Language Models
• RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratch
• The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games
• Honey, I Shrunk The Actor: A Case Study on Preserving Performance with Smaller Actors in Actor-Critic RL
• Randomized Ensembled Double Q-Learning: Learning Fast Without a Model
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep\n Reinforcement Learning
• Array programming with NumPy
• Understanding the impact of entropy on policy optimization
• Proximal Policy Optimization Algorithms
• Deep Exploration via Bootstrapped DQN
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization

Cited by

• Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization
• ReDiPPO: Reference-Guided Value Calibration and Discrepancy-Aware Token Reweighting for Mathematical Reasoning

Related

• APO: Enhancing Reasoning Ability of MLLMs via Asymmetric Policy Optimization
• Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment
• Truncated Proximal Policy Optimization
• Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO
• Unearthing Gems from Stones: Policy Optimization with Negative Sample Augmentation for LLM Reasoning