Instant Policy: In-Context Imitation Learning via Graph Diffusion

2024/11/19 by Vitalis Vosylius, Vosylius, Vitalis, Edward Johns +1 · 14 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Robotics (cs.RO) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2411.12633

openalex publication_date 2024/11/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Following the impressive capabilities of in-context learning with large transformers, In-Context Imitation Learning (ICIL) is a promising opportunity for robotics. We introduce Instant Policy, which learns new tasks instantly (without further training) from just one or two demonstrations, achieving ICIL through two key components. First, we introduce inductive biases through a graph representation and model ICIL as a graph generation problem with a learned diffusion process, enabling structured reasoning over demonstrations, observations, and actions. Second, we show that such a model can be trained using pseudo-demonstrations - arbitrary trajectories generated in simulation - as a virtually infinite pool of training data. Simulated and real experiments show that Instant Policy enables rapid learning of various everyday robot tasks. We also show how it can serve as a foundation for cross-embodiment and zero-shot transfer to language-defined tasks. Code and videos are available at https://www.robot-learning.uk/instant-policy.

Cited by

• Try Once, Then Optimal: De-Redundified Procedure Memory for Cross-Episode Exploration Amortization
• OMP: One-step Meanflow Policy with Directional Alignment
• ManiLong-Shot: Interaction-Aware One-Shot Imitation Learning for Long-Horizon Manipulation
• Generalizable Insights for Graph Transformers in Theory and Practice
• X-Diffusion: Training Diffusion Policies on Cross-Embodiment Human Demonstrations
• Learning Affordances at Inference-Time for Vision-Language-Action Models
• MaNGO - Adaptable Graph Network Simulators via Meta-Learning
• Annotation-Free One-Shot Imitation Learning for Multi-Step Manipulation Tasks
• IA-VLA: Input Augmentation for Vision-Language-Action models in settings with semantically complex tasks
• Large Pre-Trained Models for Bimanual Manipulation in 3D
• History-Aware Visuomotor Policy Learning via Point Tracking
• Variational Shape Inference for Grasp Diffusion on SE(3)
• MimicDroid: In-Context Learning for Humanoid Robot Manipulation from Human Play Videos
• Learning in ImaginationLand: Omnidirectional Policies through 3D Generative Models (OP-Gen)

Related

• Robust Instant Policy: Leveraging Student's t-Regression Model for Robust In-context Imitation Learning of Robot Manipulation
• Unpacking the Individual Components of Diffusion Policy
• Divide & Conquer Imitation Learning
• Few-Shot In-Context Imitation Learning via Implicit Graph Alignment
• ICLR: In-Context Imitation Learning with Visual Reasoning