Visuo-Tactile Transformers for Manipulation

2022/09/30 by Yizhou Chen, Andrea Sipos, Chen, Yizhou +5 · 12 citations
Neuroscience · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural dynamics and brain function #Neuroscience and Neural Engineering #Robotics (cs.RO) #Tactile and Sensory Interactions

paper · pdf · doi:10.48550/arxiv.2210.00121

openalex publication_date 2022/09/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Learning representations in the joint domain of vision and touch can improve manipulation dexterity, robustness, and sample-complexity by exploiting mutual information and complementary cues. Here, we present Visuo-Tactile Transformers (VTTs), a novel multimodal representation learning approach suited for model-based reinforcement learning and planning. Our approach extends the Visual Transformer \citedosovitskiy2021image to handle visuo-tactile feedback. Specifically, VTT uses tactile feedback together with self and cross-modal attention to build latent heatmap representations that focus attention on important task features in the visual domain. We demonstrate the efficacy of VTT for representation learning with a comparative evaluation against baselines on four simulated robot tasks and one real world block pushing task. We conduct an ablation study over the components of VTT to highlight the importance of cross-modality in representation learning.

Cited by

• N₀-VTLA: Scaling Vision-Tactile-Language-Action Model with Latent Tactile Tokens
• Self-Supervised Multisensory Pretraining for Contact-Rich Robot Reinforcement Learning
• Multimodal Diffusion Forcing for Forceful Manipulation
• Enhancing Tactile-based Reinforcement Learning for Robotic Control
• Spatially anchored Tactile Awareness for Robust Dexterous Manipulation
• ViTacGen: Robotic Pushing with Vision-to-Touch Generation
• Multi-Modal Manipulation via Multi-Modal Policy Consensus
• TA-VLA: Elucidating the Design Space of Torque-aware Vision-Language-Action Models
• Surformer v2: A Multimodal Classifier for Surface Understanding from Touch and Vision
• Tactile Robotics: An Outlook
• Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper
• ConViTac: Aligning Visual-Tactile Fusion with Contrastive Representations

Related

• The Power of the Senses: Generalizable Manipulation from Vision and Touch through Masked Multimodal Learning
• On the Importance of Tactile Sensing for Imitation Learning: A Case Study on Robotic Match Lighting
• ViTaPEs: Visuotactile Position Encodings for Cross-Modal Alignment in Multimodal Transformers
• Multimodal Visual-Tactile Representation Learning through Self-Supervised Contrastive Pre-Training
• Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper