From Observation to Action: Latent Action-based Primitive Segmentation for VLA Pre-training in Industrial Settings

2025/11/26 by Jiajie Zhang, Zhang, Jiajie, Sören Schwertfeger +3
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human Pose and Action Recognition #Multimodal Machine Learning Applications #Robot Manipulation and Learning #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.2511.21428

openalex publication_date 2025/11/26 · openalex created_date 2025/11/28 · openalex updated_date 2026/07/28

Abstract

We present a novel unsupervised framework to unlock vast unlabeled human demonstration data from continuous industrial video streams for Vision-Language-Action (VLA) model pre-training. Our method first trains a lightweight motion tokenizer to encode motion dynamics, then employs an unsupervised action segmenter leveraging a novel "Latent Action Energy" metric to discover and segment semantically coherent action primitives. The pipeline outputs both segmented video clips and their corresponding latent action sequences, providing structured data directly suitable for VLA pre-training. Evaluations on public benchmarks and a proprietary electric motor assembly dataset demonstrate effective segmentation of key tasks performed by humans at workstations. Further clustering and quantitative assessment via a Vision-Language Model confirm the semantic coherence of the discovered action primitives. To our knowledge, this is the first fully automated end-to-end system for extracting and organizing VLA pre-training data from unstructured industrial videos, offering a scalable solution for embodied AI integration in manufacturing.

Citations

• AMPLIFY: Actionless Motion Priors for Robot Learning from Videos
• π_0.5: a Vision-Language-Action Model with Open-World Generalization
• GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
• AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems
• Magma: A Foundation Model for Multimodal AI Agents
• Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videos
• Latent Action Pretraining from Videos
• Algorithmic Capabilities of Random Transformers
• OpenVLA: An Open-Source Vision-Language-Action Model
• Efficient Temporal Action Segmentation via Boundary-aware Query Voting
• Efficient and Effective Weakly-Supervised Action Segmentation via Action-Transition-Aware Boundary Alignment
• DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
• Learning to Act without Actions
• Open X-Embodiment: Robotic Learning Datasets and RT-X Models
• Finite Scalar Quantization: VQ-VAE Made Simple
• OTAS: Unsupervised Boundary Detection for Object-Centric Temporal Action Segmentation
• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
• RT-1: Robotics Transformer for Real-World Control at Scale
• Temporal Action Segmentation: An Analysis of Modern Techniques
• Multi-CLS BERT: An Efficient Alternative to Traditional Ensembling
• Ego4D: Around the World in 3,000 Hours of Egocentric Video
• Learning Transferable Visual Models From Natural Language Supervision
• Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
• Silhouettes: A graphical aid to the interpretation and validation of cluster analysis
• A Threshold Selection Method from Gray-Level Histograms

Related

• Latent Action Pretraining from Videos
• RynnVLA-001: Using Human Demonstrations to Improve Robot Manipulation
• Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videos
• Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videos
• Latent Action Pretraining Through World Modeling