Generative Actor-Critic: An Off-policy Algorithm Using the Push-forward Model

2021/05/08 by Lingwei Peng, Peng, Lingwei, Qian, Hui +3
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2105.03733

openalex publication_date 2021/05/08 · openalex created_date 2021/05/24 · openalex updated_date 2026/07/28

Abstract

Model-free deep reinforcement learning has achieved great success in many domains, such as video games, recommendation systems and robotic control tasks. In continuous control tasks, widely used policies with Gaussian distributions results in ineffective exploration of environments and limited performance of algorithms in many cases. In this paper, we propose a density-free off-policy algorithm, Generative Actor-Critic(GAC), using the push-forward model to increase the expressiveness of policies, which also includes an entropy-like technique, MMD-entropy regularizer, to balance the exploration and exploitation. Additionnally, we devise an adaptive mechanism to automatically scale this regularizer, which further improves the stability and robustness of GAC. The experiment results show that push-forward policies possess desirable features, such as multi-modality, which can improve the efficiency of exploration and asymptotic performance of algorithms obviously.

Citations

• Playing Atari with Deep Reinforcement Learning
• Continuous control with deep reinforcement learning
• Equivalence Between Policy Gradients and Soft Q-Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Addressing Function Approximation Error in Actor-Critic Methods
• Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research

Related

• Off-Policy Actor-Critic in an Ensemble: Achieving Maximum General Entropy and Effective Environment Exploration in Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Behavior-Guided Actor-Critic: Improving Exploration via Learning Policy Behavior Representation for Deep Reinforcement Learning
• Effective Reinforcement Learning Control using Conservative Soft Actor-Critic
• Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policies