Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning

2019/12/23 by Richard Li, Li, Richard, Allan Jabri +5 · 21 citations
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Artificial intelligence #Block (permutation group theory) #Computer science #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Generalization #Human–computer interaction #Inductive logic programming #Machine Learning (cs.LG) #Machine learning #Object (grammar) #Programming language #Reinforcement Learning in Robotics #Reinforcement learning #Robot Manipulation and Learning #Robotics (cs.RO) #Scratch #Task (project management) #cs.AI #cs.LG #cs.RO

paper · pdf · doi:10.48550/arxiv.1912.11032

published in arXiv (Cornell University) (Cornell University) · 10 pages, 4 figures and 1 table in main article, 3 figures and 3 tables in appendix. Supplementary website and videos at https://richardrl.github.io/relational-rl/

arxiv created 2019/12/23 · openalex publication_date 2019/12/23 · arxiv updated 2019/12/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Learning robotic manipulation tasks using reinforcement learning with sparse rewards is currently impractical due to the outrageous data requirements. Many practical tasks require manipulation of multiple objects, and the complexity of such tasks increases with the number of objects. Learning from a curriculum of increasingly complex tasks appears to be a natural solution, but unfortunately, does not work for many scenarios. We hypothesize that the inability of the state-of-the-art algorithms to effectively utilize a task curriculum stems from the absence of inductive biases for transferring knowledge from simpler to complex tasks. We show that graph-based relational architectures overcome this limitation and enable learning of complex tasks when provided with a simple curriculum of tasks with increasing numbers of objects. We demonstrate the utility of our framework on a simulated block stacking task. Starting from scratch, our agent learns to stack six blocks into a tower. Despite using step-wise sparse rewards, our method is orders of magnitude more data-efficient and outperforms the existing state-of-the-art method that utilizes human demonstrations. Furthermore, the learned policy exhibits zero-shot generalization, successfully stacking blocks into taller towers and previously unseen configurations such as pyramids, without any further training.

Citations

• Learning to Execute
• Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
• Artificial intelligence: a modern approach
• Learning and development in neural networks: the importance of starting small
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Continuous control with deep reinforcement learning
• Semi-Supervised Classification with Graph Convolutional Networks
• Interaction Networks for Learning about Objects, Relations and Physics
• Data-efficient Deep Reinforcement Learning for Dexterous Manipulation
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal\n Networks
• Automatic Goal Generation for Reinforcement Learning Agents
• Reverse Curriculum Generation for Reinforcement Learning
• Divide-and-Conquer Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
• On First-Order Meta-Learning Algorithms
• A Study on Overfitting in Deep Reinforcement Learning
• Quantifying Generalization in Reinforcement Learning
• Modulated Policy Hierarchies
• Learning to Control Self-Assembling Morphologies: A Study of Generalization via Modularity
• Rainbow: Combining Improvements in Deep Reinforcement Learning

Cited by

• RESTL: Reinforcement Learning Guided by Multi-Aspect Rewards for Signal Temporal Logic Transformation
• Learning Interactive World Model for Object-Centric Reinforcement Learning
• Flexible and Efficient Long-Range Planning Through Curious Exploration
• Efficient and Interpretable Robot Manipulation with Graph Neural Networks
• Counterfactual Data Augmentation using Locally Factored Dynamics
• XPG-RL: Reinforcement Learning with Explainable Priority Guidance for Efficiency-Boosted Mechanical Search
• Learning Compositional Neural Programs for Continuous Control

Related

• Compositional Multi-Object Reinforcement Learning with Linear Relation Networks
• Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapes
• CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and\n Transfer Learning
• COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning
• CRAFT: Coaching Reinforcement Learning Autonomously using Foundation Models for Multi-Robot Coordination Tasks