Tactile-WAM: Touch-Aware World Action Model with Tactile Asymmetric Attention

2026/06/30 by Siyu Wu, Linjing You, Junjie Zhu +9
Computer Science · #cs.RO

paper · pdf

Submitted to RSS2026 WorkShop Tactile for FM

arxiv created 2026/08/06 · arxiv updated 2026/08/07

Abstract

World Action Models (WAMs) jointly predict future visual observations and actions, but visual futures alone often miss slip, jamming, contact-direction changes, and subtle misalign- ment in contact-rich manipulation. Tactile signals reveal these hidden physical states, yet naive tactile-token injection can disrupt visual dynamics modeling due to the limited scale of tactile data, a phenomenon we term tactile pollution. We in- troduce Tactile-WAM, which uses asymmetric attention to block video queries from tactile keys while preserving tac- tile access for action queries. A contact-change-aware bias further strengthens action attention to touch. Because tactile pixel changes do not reliably reflect contact changes, we derive Observed proxy changes drive the attention bias, while future- proxy supervision preserves action-relevant contact dynamics in predicted tactile representations. On ManiFeel, visual-path isolation reduces deviation from the RGB-only trajectory by 21.8% in MSE at the step-matched 20K checkpoint without a statistically detectable change in ground-truth video qual- ity. The full model improves average success from 15.6% to 32.7%, with VideoClean providing the largest gain. On five real-robot tasks, Tactile-WAM achieves 49.2% success.

Citations

• Native Video-Action Pretraining for Generalizable Robot Control
• World Action Models are Zero-shot Policies
• ManiLong-Shot: Interaction-Aware One-Shot Imitation Learning for Long-Horizon Manipulation
• TouchFormer: A Robust Transformer-based Framework for Multimodal Material Perception
• H-GAR: A Hierarchical Interaction Framework via Goal-Driven Observation-Action Refinement for Robotic Manipulation
• Gentle Manipulation Policy Learning via Demonstrations from VLM Planned Atomic Skills
• ReconVLA: Reconstructive Vision-Language-Action Model as Effective Robot Perceiver
• MP1: MeanFlow Tames Policy Learning in 1-step for Robotic Manipulation
• ManiFeel: Benchmarking and Understanding Visuotactile Manipulation Policy Learning
• Wan: Open and Advanced Large-Scale Video Generative Models
• Reactive Diffusion Policy: Slow-Fast Visual-Tactile Policy Learning for Contact-Rich Manipulation
• Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations
• Prediction with Action: Visual Policy Learning via Joint Denoising Process
• Sparsh: Self-supervised touch representations for vision-based tactile sensing
• FoAM: Foresight-Augmented Multi-Task Imitation Policy for Robotic Manipulation
• UniT: Data Efficient Tactile Representation with Generalization to Unseen Objects
• DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation