Self-Imitation Learning for Robot Tasks with Sparse and Delayed Rewards

2020/10/14 by Zhixin Chen, Chen, Zhixin, Mengxiang Lin +1 · 1 citation
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #Robot Manipulation and Learning #Robotic Locomotion and Control

paper · pdf · doi:10.48550/arxiv.2010.06962

openalex publication_date 2020/10/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The application of reinforcement learning (RL) in robotic control is still limited in the environments with sparse and delayed rewards. In this paper, we propose a practical self-imitation learning method named Self-Imitation Learning with Constant Reward (SILCR). Instead of requiring hand-defined immediate rewards from environments, our method assigns the immediate rewards at each timestep with constant values according to their final episodic rewards. In this way, even if the dense rewards from environments are unavailable, every action taken by the agents would be guided properly. We demonstrate the effectiveness of our method in some challenging continuous robotics control tasks in MuJoCo simulation and the results show that our method significantly outperforms the alternative methods in tasks with sparse and delayed rewards. Even compared with alternatives with dense rewards available, our method achieves competitive performance. The ablation experiments also show the stability and reproducibility of our method.

Citations

• Benchmarking Deep Reinforcement Learning for Continuous Control
• Generative Adversarial Imitation Learning
• Model-Free Episodic Control
• Neural Episodic Control
• Distributed Prioritized Experience Replay
• Learning Self-Imitating Diverse Policies
• Exploration by Random Network Distillation
• Learning agile and dynamic motor skills for legged robots
• Deep Q-learning from Demonstrations
• SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards
• Dota 2 with Large Scale Deep Reinforcement Learning
• Soft Hindsight Experience Replay

Cited by

• Attention-Based Reward Shaping for Sparse and Delayed Rewards

Related

• Episodic Self-Imitation Learning with Hindsight
• Learning Sparse Rewarded Tasks from Sub-Optimal Demonstrations
• Scalable Multi-Task Imitation Learning with Autonomous Improvement
• Delayed Reinforcement Learning by Imitation
• Accelerating Self-Imitation Learning from Demonstrations via Policy Constraints and Q-Ensemble