Attraction-Repulsion Actor-Critic for Continuous Control Reinforcement Learning

2019/09/17 by Thang Doan, Doan, Thang, Bogdan Mazoure +10
Computer Science · Mathematics · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Multiagent Systems (cs.MA) #Reinforcement Learning in Robotics #cs.AI #cs.LG #cs.MA #stat.ML

paper · pdf · doi:10.48550/arxiv.1909.07543

openalex publication_date 2019/09/17 · openalex created_date 2019/09/26 · arxiv created 2020/07/09 · arxiv updated 2020/07/10 · openalex updated_date 2026/07/28

Abstract

Continuous control tasks in reinforcement learning are important because they provide an important framework for learning in high-dimensional state spaces with deceptive rewards, where the agent can easily become trapped into suboptimal solutions. One way to avoid local optima is to use a population of agents to ensure coverage of the policy space, yet learning a population with the "best" coverage is still an open problem. In this work, we present a novel approach to population-based RL in continuous control that leverages properties of normalizing flows to perform attractive and repulsive operations between current members of the population and previously observed policies. Empirical results on the MuJoCo suite demonstrate a high performance gain for our algorithm compared to prior work, including Soft-Actor Critic (SAC).

Citations

• Trust Region Policy Optimization
• A Markovian Decision Process
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Bayesian Policy Gradients via Alpha Divergence Dropout Inference
• Meta-Reinforcement Learning of Structured Exploration Strategies
• Boosting Trust Region Policy Optimization by Normalizing Flows Policy
• Evolution-Guided Policy Gradient in Reinforcement Learning
• Variational Inference with Normalizing Flows
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Related

• Solving Continuous Control via Q-learning
• Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policies
• Computationally Efficient Reinforcement Learning: Targeted Exploration leveraging Simple Rules
• Deep RL With Information Constrained Policies: Generalization in Continuous Control
• Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learning