Cautious Actor-Critic

2021/07/12 by Lingwei Zhu, Zhu, Lingwei, Toshinori Kitamura +3
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.2107.05217

Accepted by Asian Conference on Machine Learning (ACML) 2021 as long oral presentation

arxiv created 2021/10/05 · arxiv updated 2021/10/06

Abstract

The oscillating performance of off-policy learning and persisting errors in the actor-critic (AC) setting call for algorithms that can conservatively learn to suit the stability-critical applications better. In this paper, we propose a novel off-policy AC algorithm cautious actor-critic (CAC). The name cautious comes from the doubly conservative nature that we exploit the classic policy interpolation from conservative policy iteration for the actor and the entropy-regularization of conservative value iteration for the critic. Our key observation is the entropy-regularized critic facilitates and simplifies the unwieldy interpolated actor update while still ensuring robust policy improvement. We compare CAC to state-of-the-art AC methods on a set of challenging continuous control problems and demonstrate that CAC achieves comparable performance while significantly stabilizes learning.

Related

• Improving Actor-Critic Training with Steerable Action-Value Approximation Errors
• Greedy Actor-Critic: A New Conditional Cross-Entropy Method for Policy Improvement
• Effective Reinforcement Learning Control using Conservative Soft Actor-Critic
• Generative Actor-Critic: An Off-policy Algorithm Using the Push-forward Model
• Actor-critic is implicitly biased towards high entropy optimal policies