ACE: An Actor Ensemble Algorithm for Continuous Control with Tree Search

2018/11/06 by Shangtong Zhang, Hao Chen, Zhang, Shangtong +3 · 2 citations
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.1811.02696

AAAI 2019

arxiv created 2018/11/06 · arxiv updated 2018/11/12

Abstract

In this paper, we propose an actor ensemble algorithm, named ACE, for continuous control with a deterministic policy in reinforcement learning. In ACE, we use actor ensemble (i.e., multiple actors) to search the global maxima of the critic. Besides the ensemble perspective, we also formulate ACE in the option framework by extending the option-critic architecture with deterministic intra-option policies, revealing a relationship between ensemble and options. Furthermore, we perform a look-ahead tree search with those actors and a learned value prediction model, resulting in a refined value estimation. We demonstrate a significant performance boost of ACE over DDPG and its variants in challenging physical robot simulators.

Cited by

• Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategies
• Wasserstein Barycenter Soft Actor-Critic

Related

• Mean Actor Critic
• Learning to Run with Actor-Critic Ensemble
• Continuous Control With Ensemble Deep Deterministic Policy Gradients
• Towards Automatic Actor-Critic Solutions to Continuous Control
• Ensemble Reinforcement Learning in Continuous Spaces -- A Hierarchical Multi-Step Approach for Policy Training