Learning Active Task-Oriented Exploration Policies for Bridging the Sim-to-Real Gap

2020/06/02 by Jacky Liang, Liang, Jacky, Saumya Saxena +3 · 5 citations
Computer Science · Engineering · #FOS: Computer and information sciences #Machine Learning and Algorithms #Reinforcement Learning in Robotics #Robot Manipulation and Learning #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.2006.01952

openalex publication_date 2020/06/02 · openalex created_date 2020/06/12 · openalex updated_date 2026/07/28

Abstract

Training robotic policies in simulation suffers from the sim-to-real gap, as simulated dynamics can be different from real-world dynamics. Past works tackled this problem through domain randomization and online system-identification. The former is sensitive to the manually-specified training distribution of dynamics parameters and can result in behaviors that are overly conservative. The latter requires learning policies that concurrently perform the task and generate useful trajectories for system identification. In this work, we propose and analyze a framework for learning exploration policies that explicitly perform task-oriented exploration actions to identify task-relevant system parameters. These parameters are then used by model-based trajectory optimization algorithms to perform the task in the real world. We instantiate the framework in simulation with the Linear Quadratic Regulator as well as in the real world with pouring and object dragging tasks. Experiments show that task-oriented exploration helps model-based policies adapt to systems with initially unknown parameters, and it leads to better task performance than task-agnostic exploration.

Citations

• GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning
• Estimating Mass Distribution of Articulated Objects using Non-prehensile Manipulation

Cited by

• Heuristic Adaptation of Potentially Misspecified Domain Support for Likelihood-Free Inference in Stochastic Dynamical Systems
• The Reality Gap in Robotics: Challenges, Solutions, and Best Practices
• Poke and Strike: Learning Task-Informed Exploration Policies
• Sampling-Based System Identification with Active Exploration for Legged Robot Sim2Real Learning
• DROID: Minimizing the Reality Gap using Single-Shot Human Demonstration

Related

• Learning to Explore in Motion and Interaction Tasks
• AdaptSim: Task-Driven Simulation Adaptation for Sim-to-Real Transfer
• Policy Transfer with Strategy Optimization
• Generalizable Domain Adaptation for Sim-and-Real Policy Co-Training
• ASID: Active Exploration for System Identification in Robotic Manipulation