τ: Learning Touch-Augmented Vision-Language-Action Models from Future Visual Supervision

2026/07/27 by Ning Cheng, Jinan Xu, Wanlin Li +5
Computer Science · #cs.RO

paper · pdf

arxiv created 2026/07/30 · arxiv updated 2026/07/31

Abstract

Incorporating tactile sensing into Vision-Language-Action (VLA) models holds promise for contact-rich manipulation, where visual observations alone often fail to capture critical cues about physical interactions. However, learning informative tactile representation while effectively adapting it to pretrained VLA models remains challenging under limited task-specific data. Existing methods either focus on instantaneous contact states or model temporal interaction dynamics using 6D wrench sequences, leaving high-dimensional tactile signals underexplored. To address these challenges, we present τ, a touch-augmented VLA framework that learns an action-conditioned spatiotemporal tactile representation from future visual supervision inspired by the Joint-Embedding Predictive Architecture (JEPA), and fuses it with vision-language features for action generation. This supervision operates in latent space and is used only during training, adding no deployment overhead. We also introduce TacAura, a dataset of synchronized vision, proprioception, and vision-based tactile signals across four representative contact-rich manipulation tasks. Experiments show that τ outperforms existing models and generalizes to unseen objects and scenes, delivering improved manipulation performance and robustness.

Citations

• MLA: A Multisensory Language-Action Model for Multimodal Understanding and Forecasting in Robotic Manipulation
• OmniVTLA: Vision-Tactile-Language-Action Model with Semantic-Aligned Tactile Sensing
• Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper
• ForceVLA: Enhancing VLA Models with a Force-aware MoE for Contact-rich Manipulation
• VTLA: Vision-Tactile-Language-Action Model with Preference Learning for Insertion Manipulation
• π_0.5: a Vision-Language-Action Model with Open-World Generalization
• Reactive Diffusion Policy: Slow-Fast Visual-Tactile Policy Learning for Contact-Rich Manipulation
• AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors
• π₀: A Vision-Language-Action Flow Model for General Robot Control
• OpenVLA: An Open-Source Vision-Language-Action Model
• Binding Touch to Everything: Learning Unified Multimodal Tactile Representations
• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
• Learn from Incomplete Tactile Data: Tactile Representation Learning with Masked Autoencoders
• Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware
• Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
• Diffusion policy: Visuomotor policy learning via action diffusion
• Self-Supervised Visuo-Tactile Pretraining to Locate and Follow Garment Features
• Learning rich touch representations through cross-modal self-supervision
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• Contrastive Multiview Coding
• Ambient Sound Provides Supervision for Visual Learning
• Zero-Shot Learning Through Cross-Modal Transfer
• Tactile-VLA: Unlocking Vision-Language-Action Model's Physical Knowledge for Tactile Generalization

Related

• Representation-Aligned Tactile Grounding for Contact-Rich Robotic Manipulation
• TLA: Tactile-Language-Action Model for Contact-Rich Manipulation
• N₀-VTLA: Scaling Vision-Tactile-Language-Action Model with Latent Tactile Tokens
• Tactile-VLA: Unlocking Vision-Language-Action Model's Physical Knowledge for Tactile Generalization
• OmniVTLA: Vision-Tactile-Language-Action Model with Semantic-Aligned Tactile Sensing