Boosting the Actor with Dual Critic

2017/12/29 by Bo Dai, Albert Shaw, Dai, Bo +7 · 2 citations
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #FOS: Computer and information sciences #Machine Learning (cs.LG) #Metaheuristic Optimization Algorithms Research #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.1712.10282

openalex publication_date 2017/12/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper proposes a new actor-critic-style algorithm called Dual Actor-Critic or Dual-AC. It is derived in a principled way from the Lagrangian dual form of the Bellman optimality equation, which can be viewed as a two-player game between the actor and a critic-like function, which is named as dual critic. Compared to its actor-critic relatives, Dual-AC has the desired property that the actor and dual critic are updated cooperatively to optimize the same objective function, providing a more transparent way for learning the critic that is directly related to the objective function of the actor. We then provide a concrete algorithm that can effectively solve the minimax optimization problem, using techniques of multi-step bootstrapping, path regularization, and stochastic dual ascent algorithm. We demonstrate that the proposed algorithm achieves the state-of-the-art performances across several benchmarks.

Cited by

• A Two-Timescale Primal-Dual Framework for Reinforcement Learning via Online Dual Variable Guidance
• Multi-Agent Reinforcement Learning via Double Averaging Primal-Dual Optimization

Related

• Variational Actor-Critic Algorithms
• Finite Time Analysis of Constrained Natural Critic-Actor Algorithm with Improved Sample Complexity
• Two-Timescale Critic-Actor for Average Reward MDPs with Function Approximation
• Provably Efficient Convergence of Primal-Dual Actor-Critic with Nonlinear Function Approximation
• Stackelberg Actor-Critic: Game-Theoretic Reinforcement Learning Algorithms