2021/05/07 by Taisei Hashimoto, Hashimoto, Taisei, Yoshimasa Tsuruoka +1
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Human Pose and Action Recognition #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Reinforcement Learning in Robotics #cs.AI #cs.LG
paper · pdf · doi:10.48550/arxiv.2105.03041
Deep Reinforcement Learning Workshop, NeurIPS 2020
arxiv created 2021/05/07 · openalex publication_date 2021/05/07 · arxiv updated 2021/05/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
In many deep reinforcement learning settings, when an agent takes an action, it repeats the same action a predefined number of times without observing the states until the next action-decision point. This technique of action repetition has several merits in training the agent, but the data between action-decision points (i.e., intermediate frames) are, in effect, discarded. Since the amount of training data is inversely proportional to the interval of action repeats, they can have a negative impact on the sample efficiency of training. In this paper, we propose a simple but effective approach to alleviate to this problem by introducing the concept of pseudo-actions. The key idea of our method is making the transition between action-decision points usable as training data by considering pseudo-actions. Pseudo-actions for continuous control tasks are obtained as the average of the action sequence straddling an action-decision point. For discrete control tasks, pseudo-actions are computed from learned action embeddings. This method can be combined with any model-free reinforcement learning algorithm that involves the learning of Q-functions. We demonstrate the effectiveness of our approach on both continuous and discrete control tasks in OpenAI Gym.