Latent Adaptive Planner for Dynamic Manipulation

2025/05/06 by Donghun Noh, Noh, Donghun, Deqian Kong +11 · 1 citation
Computer Science · Engineering · #AI-based Problem Solving and Planning #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Human Motion and Animation #Machine Learning (cs.LG) #Robot Manipulation and Learning #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.2505.03077

openalex publication_date 2025/05/06 · openalex created_date 2025/10/16 · openalex updated_date 2026/07/28

Abstract

We present the Latent Adaptive Planner (LAP), a trajectory-level latent-variable policy for dynamic nonprehensile manipulation (e.g., box catching) that formulates planning as inference in a low-dimensional latent space and is learned effectively from human demonstration videos. During execution, LAP achieves real-time adaptation by maintaining a posterior over the latent plan and performing variational replanning as new observations arrive. To bridge the embodiment gap between humans and robots, we introduce a model-based proportional mapping that regenerates accurate kinematic-dynamic joint states and object positions from human demonstrations. Through challenging box catching experiments with varying object properties, LAP demonstrates superior success rates, trajectory smoothness, and energy efficiency by learning human-like compliant motions and adaptive behaviors. Overall, LAP enables dynamic manipulation with real-time adaptation and successfully transfer across heterogeneous robot platforms using the same human demonstration videos.

Citations

• Phantom: Training Robots Without Robots Using Only Human Videos
• Latent Thought Models with Variational Bayes Inference-Time Computation
• Latent Space Energy-based Neural ODEs
• RoVi-Aug: Robot and Viewpoint Augmentation for Cross-Embodiment Robot Learning
• Impact-Aware Bimanual Catching of Large-Momentum Objects
• DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
• Latent Plan Transformer for Trajectory Abstraction: Planning as Latent Space Inference
• Molecule Design by Latent Prompt Transformer
• Molecule Design by Latent Prompt Transformer
• Deep Imitation Learning for Humanoid Loco-manipulation through Human Teleoperation
• Diverse and Faithful Knowledge-Grounded Dialogue Generation via Sequential Posterior Inference
• Legged Robots for Object Manipulation: A Review
• Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
• Diffusion policy: Visuomotor policy learning via action diffusion
• Teleoperation of Humanoid Robots: A Survey
• Dichotomy of Control: Separating What You Can Control from What You Cannot
• Latent Diffusion Energy-Based Model for Interpretable Text Modeling
• You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environments
• Planning with Diffusion for Flexible Behavior Synthesis
• Iterative Residual Policy: for Goal-Conditioned Dynamic Manipulation of Deformable Objects
• DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Video
• Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets
• CLIPort: What and Where Pathways for Robotic Manipulation
• DexMV: Imitation Learning for Dexterous Manipulation from Human Videos
• Offline Reinforcement Learning as One Big Sequence Modeling Problem
• Decision Transformer: Reinforcement Learning via Sequence Modeling
• OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning
• Visual Imitation Made Easy
• Learning Latent Space Energy-Based Prior Model
• Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrations
• Learning Predictive Models From Observation and Interaction
• IRIS: Implicit Reinforcement without Interaction at Scale for Learning\n Control from Offline Robot Manipulation Data
• TossingBot: Learning to Throw Arbitrary Objects with Residual Physics
• Learning Latent Plans from Play
• Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data\n for Imitation
• Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation
• Learning Complex Dexterous Manipulation with Deep Reinforcement Learning\n and Demonstrations
• Attention Is All You Need
• Variational Inference: A Review for Statisticians
• A survey of robot learning from demonstration

Cited by

• Prior Reinforce: Mastering Agile Tasks with Limited Trials

Related

• Pixels to Plans: Learning Non-Prehensile Manipulation by Imitating a Planner
• Jacta: A Versatile Planner for Learning Dexterous and Whole-body Manipulation
• DynaMimicGen: A Data Generation Framework for Robot Learning of Dynamic Tasks
• Dynamics Learning with Cascaded Variational Inference for Multi-Step Manipulation
• Decompose and Reorganize: Planning with Primitives and Visuomotor Policies Learned from Demonstrations