Reinforcement Learning via Conservative Agent for Environments with Random Delays

2025/07/25 by Jongsoo Lee, Jang Won Kim, Lee, Jongsoo +5
Computer Science · #Adaptive Dynamic Programming Control #Age of Information Optimization #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2507.18992

openalex publication_date 2025/07/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Real-world reinforcement learning applications are often hindered by delayed feedback from environments, which violates the Markov assumption and introduces significant challenges. Although numerous delay-compensating methods have been proposed for environments with constant delays, environments with random delays remain largely unexplored due to their inherent variability and unpredictability. In this study, we propose a simple yet robust agent for decision-making under random delays, termed the conservative agent, which reformulates the random-delay environment into its constant-delay equivalent. This transformation enables any state-of-the-art constant-delay method to be directly extended to the random-delay environments without modifying the algorithmic structure or sacrificing performance. We evaluate the conservative agent-based algorithm on continuous control tasks, and empirical results demonstrate that it significantly outperforms existing baseline algorithms in terms of asymptotic performance and sample efficiency.

Citations

• Humanoid Parkour Learning
• Variational Delayed Policy Optimization
• Boosting Reinforcement Learning with Strongly Delayed Feedback Through Auxiliary Short Delays
• Delays in Reinforcement Learning
• Robot Parkour Learning
• ANYmal Parkour: Learning Agile Navigation for Quadrupedal Robots
• Delayed Reinforcement Learning by Imitation
• Magnetic control of tokamak plasmas through deep reinforcement learning
• Acting in Delayed Environments with Non-Stationary Markov Policies
• SCC: an efficient deep reinforcement learning agent mastering the game of StarCraft II
• Reinforcement Learning with Random Delays
• Language Models are Few-Shot Learners
• Soft Actor-Critic Algorithms and Applications
• At Human Speed: Deep Reinforcement Learning with Action Delay
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Setting up a Reinforcement Learning Task with a Real-World Robot
• Addressing Function Approximation Error in Actor-Critic Methods
• Adam: A Method for Stochastic Optimization
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Playing Atari with Deep Reinforcement Learning

Related

• Reinforcement Learning with Random Delays
• Adaptive Reinforcement Learning for Unobservable Random Delays
• Delays in Reinforcement Learning
• Boosting Reinforcement Learning with Strongly Delayed Feedback Through Auxiliary Short Delays
• Rainbow Delay Compensation: A Multi-Agent Reinforcement Learning Framework for Mitigating Delayed Observation