TAAC: Temporally Abstract Actor-Critic for Continuous Control

2021/04/13 by Haonan Yu, Wei Xu, Yu, Haonan +3
Computer Science · Decision Sciences · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Formal Methods in Verification #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #Robotics (cs.RO) #Simulation Techniques and Applications

paper · pdf · doi:10.48550/arxiv.2104.06521

openalex publication_date 2021/04/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present temporally abstract actor-critic (TAAC), a simple but effective off-policy RL algorithm that incorporates closed-loop temporal abstraction into the actor-critic framework. TAAC adds a second-stage binary policy to choose between the previous action and a new action output by an actor. Crucially, its "act-or-repeat" decision hinges on the actually sampled action instead of the expected behavior of the actor. This post-acting switching scheme let the overall policy make more informed decisions. TAAC has two important features: a) persistent exploration, and b) a new compare-through Q operator for multi-step TD backup, specially tailored to the action repetition scenario. We demonstrate TAAC's advantages over several strong baselines across 14 continuous control tasks. Our surprising finding reveals that while achieving top performance, TAAC is able to "mine" a significant number of repeated actions with the trained policy even on continuous tasks whose problem structures on the surface seem to repel action repetition. This suggests that aside from encouraging persistent exploration, action repetition can find its place in a good policy behavior. Code is available at https://github.com/hnyu/taac.

Citations

• Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
• FeUdal Networks for Hierarchical Reinforcement Learning
• Hindsight Experience Replay
• Multi-Goal Reinforcement Learning: Challenging Robotics Environments and\n Request for Research
• Data-Efficient Hierarchical Reinforcement Learning
• Relative Entropy Regularized Policy Iteration
• Hybrid Actor-Critic Reinforcement Learning in Parameterized Action Space
• Learning Multi-Level Hierarchies with Hindsight
• Discrete and Continuous Action Representation for Practical RL in Video Games
• Hierarchical Reinforcement Learning By Discovering Intrinsic Options

Related

• How to Learn a Useful Critic? Model-based Action-Gradient-Estimator Policy Optimization
• AFU: Actor-Free critic Updates in off-policy RL for continuous control
• Actor-Critic with variable time discretization via sustained actions
• Learning to Repeat: Fine Grained Action Repetition for Deep\n Reinforcement Learning
• Online Meta-Critic Learning for Off-Policy Actor-Critic Methods