Actionable Models: Unsupervised Offline Reinforcement Learning of\n Robotic Skills

2021/04/15 by Yevgen Chebotar, Karol Hausman, Chebotar, Yevgen +20 · 10 citations
Computer Science · #Multimodal Machine Learning Applications #Domain Adaptation and Few-Shot Learning #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2104.07749

Abstract

We consider the problem of learning useful robotic skills from previously\ncollected offline data without access to manually specified rewards or\nadditional online exploration, a setting that is becoming increasingly\nimportant for scaling robot learning by reusing past robotic data. In\nparticular, we propose the objective of learning a functional understanding of\nthe environment by learning to reach any goal state in a given dataset. We\nemploy goal-conditioned Q-learning with hindsight relabeling and develop\nseveral techniques that enable training in a particularly challenging offline\nsetting. We find that our method can operate on high-dimensional camera images\nand learn a variety of skills on real robots that generalize to previously\nunseen scenes and objects. We also show that our method can learn to reach\nlong-horizon goals across multiple episodes through goal chaining, and learn\nrich representations that can help with downstream tasks through pre-training\nor auxiliary objectives. The videos of our experiments can be found at\nhttps://actionable-models.github.io\n

Citations

• Successor Features for Transfer in Reinforcement Learning
• Goal-Conditioned Imitation Learning
• VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation
• COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning

Cited by

• Action-Sufficient Goal Representations
• Goal Reaching with Eikonal-Constrained Hierarchical Quasimetric Reinforcement Learning
• The Information Geometry of Unsupervised Reinforcement Learning
• OmniVLA: Physically-Grounded Multimodal VLA with Unified Multi-Sensor Perception for Robotic Manipulation
• A Workflow for Offline Model-Free Robotic Reinforcement Learning
• A Unified Framework for Zero-Shot Reinforcement Learning
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• Self-Improving Embodied Foundation Models
• Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learning
• Visuomotor Grasping with World Models for Surgical Robots

Related

• Efficient Self-Supervised Data Collection for Offline Robot Learning
• A Workflow for Offline Model-Free Robotic Reinforcement Learning
• Don't Start From Scratch: Leveraging Prior Data to Automate Robotic Reinforcement Learning
• Latent Plans for Task-Agnostic Offline Reinforcement Learning
• Real-World Offline Reinforcement Learning from Vision Language Model Feedback