Discrete Action On-Policy Learning with Action-Value Critic

2020/02/10 by Yuguang Yue, Yunhao Tang, Yue, Yuguang +5 · 1 citation
Computer Science · Mathematics · Physics and Astronomy · #Action (physics) #Adversarial Robustness in Machine Learning #Artificial intelligence #Benchmark (surveying) #Computer science #Construct (python library) #Convergence (economics) #Dimension (graph theory) #Discretization #FOS: Computer and information sciences #Flexibility (engineering) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Mathematical optimization #Mathematics #Model Reduction and Neural Networks #Reinforcement Learning in Robotics #Reinforcement learning #Space (punctuation) #Statistics #Variance (accounting) #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2002.03534

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2020/02/10 · arxiv created 2020/02/21 · arxiv updated 2020/02/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06

Abstract

Reinforcement learning (RL) in discrete action space is ubiquitous in real-world applications, but its complexity grows exponentially with the action-space dimension, making it challenging to apply existing on-policy gradient based deep RL algorithms efficiently. To effectively operate in multidimensional discrete action spaces, we construct a critic to estimate action-value functions, apply it on correlated actions, and combine these critic estimated action values to control the variance of gradient estimation. We follow rigorous statistical analysis to design how to generate and combine these correlated actions, and how to sparsify the gradients by shutting down the contributions from certain dimensions. These efforts result in a new discrete action on-policy RL algorithm that empirically outperforms related on-policy algorithms relying on variance control techniques. We demonstrate these properties on OpenAI Gym benchmark tasks, and illustrate how discretizing the action space could benefit the exploration phase and hence facilitate convergence to a better local optimal solution thanks to the flexibility of discrete policy.

Citations

• High-Dimensional Continuous Control Using Generalized Advantage\n Estimation
• Playing Atari with Deep Reinforcement Learning
• Trust Region Policy Optimization
• Auto-Encoding Variational Bayes
• Continuous control with deep reinforcement learning
• Sample Efficient Actor-Critic with Experience Replay
• Reinforcement Learning with Deep Energy-Based Policies
• REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models
• Stein Variational Policy Gradient
• Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Addressing Function Approximation Error in Actor-Critic Methods
• Discretizing Continuous Action Space for On-Policy Optimization
• ARSM: Augment-REINFORCE-Swap-Merge Estimator for Gradient Backpropagation Through Categorical Variables
• Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
• Learning dexterous in-hand manipulation
• Variational Inference: A Review for Statisticians

Cited by

• Who Gets the Reward & Who Gets the Blame? Evaluation-Aligned Training Signals for Multi-LLM Agents

Related

• Mean Actor Critic
• Dissecting Discrete Soft Actor-Critic: Limitations and Principled Alternatives
• Soft Decomposed Policy-Critic: Bridging the Gap for Effective Continuous Control with Discrete RL
• Convergent Actor-Critic Algorithms Under Off-Policy Training and Function Approximation
• Distributions as Actions: A Unified Framework for Diverse Action Spaces