Soft Hindsight Experience Replay

2020/02/06 by Qiwei He, He, Qiwei, Liansheng Zhuang +3 · 2 citations
Computer Science · Engineering · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Reinforcement Learning in Robotics #Robot Manipulation and Learning #Robotics (cs.RO) #cs.AI #cs.RO

paper · pdf · doi:10.48550/arxiv.2002.02089

7 pages, 5 figures, 1 table, submitted to IJCAI2020

arxiv created 2020/02/06 · openalex publication_date 2020/02/06 · arxiv updated 2020/02/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Efficient learning in the environment with sparse rewards is one of the most important challenges in Deep Reinforcement Learning (DRL). In continuous DRL environments such as robotic arms control, Hindsight Experience Replay (HER) has been shown an effective solution. However, due to the brittleness of deterministic methods, HER and its variants typically suffer from a major challenge for stability and convergence, which significantly affects the final performance. This challenge severely limits the applicability of such methods to complex real-world domains. To tackle this challenge, in this paper, we propose Soft Hindsight Experience Replay (SHER), a novel approach based on HER and Maximum Entropy Reinforcement Learning (MERL), combining the failed experiences reuse and maximum entropy probabilistic inference model. We evaluate SHER on Open AI Robotic manipulation tasks with sparse rewards. Experimental results show that, in contrast to HER and its variants, our proposed SHER achieves state-of-the-art performance, especially in the difficult HandManipulation tasks. Furthermore, our SHER method is more stable, achieving very similar performance across different random seeds.

Citations

• Playing Atari with Deep Reinforcement Learning
• Continuous control with deep reinforcement learning
• Equivalence Between Policy Gradients and Soft Q-Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• If MaxEnt RL is the Answer, What is the Question?

Cited by

• GCHR : Goal-Conditioned Hindsight Regularization for Sample-Efficient Reinforcement Learning
• Self-Imitation Learning for Robot Tasks with Sparse and Delayed Rewards

Related

• Hindsight Experience Replay
• Learning and reusing primitive behaviours to improve Hindsight Experience Replay sample efficiency
• Contact Energy Based Hindsight Experience Prioritization
• Deep Reinforcement Learning for Complex Manipulation Tasks with Sparse Feedback
• MHER: Model-based Hindsight Experience Replay