Learning from Pixels with Expert Observations

2023/06/24 by Minh-Huy Hoang, Hoang, Minh-Huy, Long Dinh +3 · 1 citation
Computer Science · Engineering · #Reinforcement Learning in Robotics #Robot Manipulation and Learning #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2306.13872

Abstract

In reinforcement learning (RL), sparse rewards can present a significant challenge. Fortunately, expert actions can be utilized to overcome this issue. However, acquiring explicit expert actions can be costly, and expert observations are often more readily available. This paper presents a new approach that uses expert observations for learning in robot manipulation tasks with sparse rewards from pixel observations. Specifically, our technique involves using expert observations as intermediate visual goals for a goal-conditioned RL agent, enabling it to complete a task by successively reaching a series of goals. We demonstrate the efficacy of our method in five challenging block construction tasks in simulation and show that when combined with two state-of-the-art agents, our approach can significantly improve their performance while requiring 4-20 times fewer expert actions during training. Moreover, our method is also superior to a hierarchical baseline.

Cited by

• Hierarchical Reinforcement Learning for Articulated Tool Manipulation with Multifingered Hand

Related

• Learning Sparse Control Tasks from Pixels by Latent Nearest-Neighbor-Guided Explorations
• Unlocking Pixels for Reinforcement Learning via Implicit Attention
• Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentation
• A Study on Dense and Sparse (Visual) Rewards in Robot Policy Learning
• DeGuV: Depth-Guided Visual Reinforcement Learning for Generalization and Interpretability in Manipulation