Policy Continuation with Hindsight Inverse Dynamics

2019/10/30 by Sun, Hao, Li, Zhizhong, Liu, Xiaotong +2 · 1 citation
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.1910.14055

Abstract

Solving goal-oriented tasks is an important but challenging problem in reinforcement learning (RL). For such tasks, the rewards are often sparse, making it difficult to learn a policy effectively. To tackle this difficulty, we propose a new approach called Policy Continuation with Hindsight Inverse Dynamics (PCHID). This approach learns from Hindsight Inverse Dynamics based on Hindsight Experience Replay, enabling the learning process in a self-imitated manner and thus can be trained with supervised learning. This work also extends it to multi-step settings with Policy Continuation. The proposed method is general, which can work in isolation or be combined with other on-policy and off-policy algorithms. On two multi-goal tasks GridWorld and FetchReach, PCHID significantly improves the sample efficiency as well as the final performance.

Cited by

• Self-Supervised Goal-Reaching Results in Multi-Agent Cooperation and Exploration

Related

• Improvements on Hindsight Learning
• Hindsight policy gradients
• MapGo: Model-Assisted Policy Optimization for Goal-Oriented Tasks
• Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement
• GCHR : Goal-Conditioned Hindsight Regularization for Sample-Efficient Reinforcement Learning