CLAM: Continuous Latent Action Models for Robot Learning from Unlabeled Demonstrations

2025/05/08 by Anthony Liang, Liang, Anthony, Pavel Czempin +11 · 12 citations
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Human Pose and Action Recognition #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #Robot Manipulation and Learning #Robotics (cs.RO) #cs.AI #cs.LG #cs.RO

paper · pdf · doi:10.48550/arxiv.2505.04999

published as IEEE/RSJ International Conference on Intelligent Robots and Systems 2026 · Latent Action Models, Self-supervised Pretraining, Learning from Videos

openalex publication_date 2025/05/08 · openalex created_date 2025/10/10 · arxiv created 2026/07/30 · arxiv updated 2026/07/31 · openalex updated_date 2026/08/02

Abstract

Learning robot control policies from demonstrations typically requires action-labeled expert data, which is expensive to collect through teleoperation. We study a more practical setting in which expert demonstrations are available only as observation sequences without action labels, and only task-agnostic play data contains actions. We introduce continuous latent action models (CLAM), a framework that infers continuous latent actions between consecutive observations using self-supervised dynamics prediction. To ground these latent actions into executable motor commands, CLAM jointly trains an action decoder using a small amount of task-agnostic play data. We show that continuous latent actions combined with this joint training are essential for high-dimensional continuous control. Across DMControl locomotion, MetaWorld manipulation, and real-world WidowX robot tasks, CLAM improves average task success rates by 2-3x over prior latent-action baselines and approaches behavior cloning trained with privileged expert action labels. Our results demonstrate that effective robot policies can be learned from unlabeled demonstrations and deployed on real hardware without collecting expert action-labeled data. Videos and code are available at clamrobot.github.io.

Citations

• Masquerade: Learning from In-the-wild Human Videos using Data-Editing
• villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models
• HAND Me the Data: Fast Robot Adaptation via Hand Path Retrieval
• CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learning
• UniVLA: Learning to Act Anywhere with Task-centric Latent Actions
• Phantom: Training Robots Without Robots Using Only Human Videos
• Object-Centric Latent Action Learning
• Latent Action Learning Requires Supervision in the Presence of Distractors
• Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation
• π₀: A Vision-Language-Action Flow Model for General Robot Control
• Latent Action Pretraining from Videos
• DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Control
• Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviation
• Multistep Inverse Is Not All You Need
• Genie: Generative Interactive Environments
• Learning to Act without Actions
• TD-MPC2: Scalable, Robust World Models for Continuous Control
• Learning to Act from Actionless Videos through Dense Correspondences
• BridgeData V2: A Dataset for Robot Learning at Scale
• Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware
• Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
• Diffusion policy: Visuomotor policy learning via action diffusion
• Learning Universal Policies via Text-Guided Video Generation
• Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos
• Temporal Difference Learning for Model Predictive Control
• CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks
• Ego4D: Around the World in 3,000 Hours of Egocentric Video
• On the Opportunities and Risks of Foundation Models
• Is Space-Time Attention All You Need for Video Understanding?
• Playable Video Generation
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Language Models are Few-Shot Learners
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Leveraging Procedural Generation to Benchmark Reinforcement Learning
• State Alignment-based Imitation Learning
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement
• Learning Latent Plans from Play
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Neural Discrete Representation Learning
• Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation

Cited by

• Embodied Robot Manipulation in the Era of Foundation Models: Planning and Learning Perspectives
• LAOF: Robust Latent Action Learning with Optical Flow Constraints
• ViPRA: Video Prediction for Robot Actions
• Learning to play: A Multimodal Agent for 3D Game-Play
• When a Robot is More Capable than a Human: Learning from Constrained Demonstrators
• CollabVLA: Self-Reflective Vision-Language-Action Model Dreaming Together with Human
• Chain-of-Action: Trajectory Autoregressive Modeling for Robotic Manipulation
• Pixels to Play: A Foundation Model for 3D Gameplay
• villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models
• CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learning
• Hierarchical Latent Action Model
• Learning Latent Action World Models In The Wild
• JoyAI-RA 0.5: Scaling Robot Manipulation Learning via Dual Action Alignment

Related

• CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learning
• Boosting Action-Information via a Variational Bottleneck on Unlabelled Robot Videos
• Action Chunking and Exploratory Data Collection Yield Exponential Improvements in Behavior Cloning for Continuous Control
• Imitation Learning with Limited Actions via Diffusion Planners and Deep Koopman Controllers
• HYDRA: Hybrid Robot Actions for Imitation Learning