Reinforcement Learning with Random Delays

2020/10/06 by Simon Ramstedt, Yann Bouteiller, Ramstedt, Simon +7 · 9 citations
Computer Science · Engineering · #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #Smart Grid Energy Management #cs.LG

paper · pdf · doi:10.48550/arxiv.2010.02966

ICLR 2021

openalex publication_date 2020/10/06 · arxiv created 2021/05/04 · arxiv updated 2021/05/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Action and observation delays commonly occur in many Reinforcement Learning applications, such as remote control scenarios. We study the anatomy of randomly delayed environments, and show that partially resampling trajectory fragments in hindsight allows for off-policy multi-step value estimation. We apply this principle to derive Delay-Correcting Actor-Critic (DCAC), an algorithm based on Soft Actor-Critic with significantly better performance in environments with delays. This is shown theoretically and also demonstrated practically on a delay-augmented version of the MuJoCo continuous control benchmark.

Citations

• Adam: A Method for Stochastic Optimization
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Addressing Function Approximation Error in Actor-Critic Methods
• Soft Actor-Critic Algorithms and Applications

Cited by

• A Survey of Freshness-Aware Wireless Networking with Reinforcement Learning
• Reinforcement Learning on Cost-Constrained Quadrupedal Hardware
• Reinforcement Learning with Action-Triggered Observations
• Model-Based Reinforcement Learning under Random Observation Delays
• Real-time reinforcement learning for turbulent state-dependent control in a bluff-body wake
• RT-HCP: Dealing with Inference Delays and Sample Efficiency to Learn Directly on Robotic Platforms
• Reinforcement Learning via Conservative Agent for Environments with Random Delays
• Rainbow Delay Compensation: A Multi-Agent Reinforcement Learning Framework for Mitigating Delayed Observation
• Belief-Based Offline Reinforcement Learning for Delay-Robust Policy Optimization

Related

• Reinforcement Learning via Conservative Agent for Environments with Random Delays
• Adaptive Reinforcement Learning for Unobservable Random Delays
• Reinforcement Learning from Delayed Observations via World Models
• Delays in Reinforcement Learning
• Model-Based Reinforcement Learning under Random Observation Delays