An Entropy Regularization Free Mechanism for Policy-based Reinforcement Learning

2021/06/01 by Changnan Xiao, Xiao, Changnan, Haosen Shi +5 · 3 citations
Computer Science · Engineering · #Adaptive Dynamic Programming Control #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #Smart Grid Energy Management

paper · pdf · doi:10.48550/arxiv.2106.00707

openalex publication_date 2021/06/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Policy-based reinforcement learning methods suffer from the policy collapse problem. We find valued-based reinforcement learning methods with ε-greedy mechanism are capable of enjoying three characteristics, Closed-form Diversity, Objective-invariant Exploration and Adaptive Trade-off, which help value-based methods avoid the policy collapse problem. However, there does not exist a parallel mechanism for policy-based methods that achieves all three characteristics. In this paper, we propose an entropy regularization free mechanism that is designed for policy-based methods, which achieves Closed-form Diversity, Objective-invariant Exploration and Adaptive Trade-off. Our experiments show that our mechanism is super sample-efficient for policy-based methods and boosts a policy-based baseline to a new State-Of-The-Art on Arcade Learning Environment.

Citations

• Long Short-Term Memory
• Dueling Network Architectures for Deep Reinforcement Learning
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Equivalence Between Policy Gradients and Soft Q-Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Soft Actor-Critic Algorithms and Applications
• Revisiting the Softmax Bellman Operator: New Benefits and New Perspective

Cited by

• Incentivizing Consistent, Effective and Scalable Reasoning Capability in Audio LLMs via Reasoning Process Rewards
• Fine-tuning Flow Matching Generative Models with Intermediate Feedback
• Entropy Meets Importance: A Unified Head Importance-Entropy Score for Stable and Efficient Transformer Pruning

Related

• Cautious Policy Programming: Exploiting KL Regularization in Monotonic Policy Improvement for Reinforcement Learning
• Implicit Policy for Reinforcement Learning
• Ensuring Monotonic Policy Improvement in Entropy-regularized Value-based Reinforcement Learning
• Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learning
• Policy Optimization Reinforcement Learning with Entropy Regularization