Leveraging Sequentiality in Reinforcement Learning from a Single Demonstration

2022/11/09 by Alexandre Chenu, Olivier Serris, Chenu, Alexandre +5
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Reinforcement Learning in Robotics #Robot Manipulation and Learning #Robotic Locomotion and Control #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.2211.04786

openalex publication_date 2022/11/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Deep Reinforcement Learning has been successfully applied to learn robotic control. However, the corresponding algorithms struggle when applied to problems where the agent is only rewarded after achieving a complex task. In this context, using demonstrations can significantly speed up the learning process, but demonstrations can be costly to acquire. In this paper, we propose to leverage a sequential bias to learn control policies for complex robotic tasks using a single demonstration. To do so, our method learns a goal-conditioned policy to control a system between successive low-dimensional goals. This sequential goal-reaching approach raises a problem of compatibility between successive goals: we need to ensure that the state resulting from reaching a goal is compatible with the achievement of the following goals. To tackle this problem, we present a new algorithm called DCIL-II. We show that DCIL-II can solve with unprecedented sample efficiency some challenging simulated tasks such as humanoid locomotion and stand-up as well as fast running with a simulated Cassie robot. Our method leveraging sequentiality is a step towards the resolution of complex robotic tasks under minimal specification effort, a key feature for the next generation of autonomous robots.

Citations

• Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
• On Curves of Minimal Length with a Constraint on Average Curvature, and with Prescribed Initial and Terminal Positions and Tangents
• Learnings Options End-to-End for Continuous Action Tasks
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Learning Montezuma's Revenge from a Single Demonstration
• Go-Explore: a New Approach for Hard-Exploration Problems
• Learning Multi-Level Hierarchies with Hindsight
• Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning
• Solving Rubik's Cube with a Robot Hand
• Primal Wasserstein Imitation Learning
• Goal-Conditioned Reinforcement Learning with Imagined Subgoals
• Reinforcement Learning: An Introduction
• Search on the Replay Buffer: Bridging Planning and Reinforcement Learning
• OpenAI Gym

Related

• Single-Reset Divide & Conquer Imitation Learning
• Overcoming Exploration in Reinforcement Learning with Demonstrations
• Lifelong Robotic Reinforcement Learning by Retaining Experiences
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Using Deep Reinforcement Learning for the Continuous Control of Robotic Arms