Connecting Touch and Vision via Cross-Modal Prediction

2019/06/14 by Yunzhu Li, Jun-Yan Zhu, Li, Yunzhu +5 · 15 citations
Computer Science · Engineering · Neuroscience · #Advanced Sensor and Energy Harvesting Materials #Computer Vision and Pattern Recognition (cs.CV) #EEG and Brain-Computer Interfaces #FOS: Computer and information sciences #Machine Learning (cs.LG) #Robotics (cs.RO) #Tactile and Sensory Interactions #cs.CV #cs.LG #cs.RO

paper · pdf · doi:10.48550/arxiv.1906.06322

Accepted to CVPR 2019. Project Page: http://visgel.csail.mit.edu/

arxiv created 2019/06/14 · openalex publication_date 2019/06/14 · arxiv updated 2019/06/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Humans perceive the world using multi-modal sensory inputs such as vision, audition, and touch. In this work, we investigate the cross-modal connection between vision and touch. The main challenge in this cross-domain modeling task lies in the significant scale discrepancy between the two: while our eyes perceive an entire visual scene at once, humans can only feel a small region of an object at any given moment. To connect vision and touch, we introduce new tasks of synthesizing plausible tactile signals from visual inputs as well as imagining how we interact with objects given tactile data as input. To accomplish our goals, we first equip robots with both visual and tactile sensors and collect a large-scale dataset of corresponding vision and tactile image sequences. To close the scale gap, we present a new conditional adversarial model that incorporates the scale and location information of the touch. Human perceptual studies demonstrate that our model can produce realistic visual images from tactile data and vice versa. Finally, we present both qualitative and quantitative experimental results regarding different system designs, as well as visualizing the learned representations of our model.

Citations

• Zero-Shot Learning by Convex Combination of Semantic Embeddings
• Hand movements: A window into haptic object recognition
• Gradient-based learning applied to document recognition
• ImageNet classification with deep convolutional neural networks
• NIPS 2016 Tutorial: Generative Adversarial Networks

Cited by

• OPENTOUCH: Bringing Full-Hand Touch to Real-World Interaction
• Phy-Tac: Toward Human-Like Grasping via Physics-Conditioned Tactile Goals
• Spatially anchored Tactile Awareness for Robust Dexterous Manipulation
• Representing Data in Robotic Tactile Perception -- A Review
• Cross-Sensor Touch Generation
• exUMI: Extensible Robot Teaching System with Action-aware Task-agnostic Tactile Representation
• Tactile Robotics: An Outlook
• Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper
• A Survey on Imitation Learning for Contact-Rich Tasks in Robotics
• Tactile MNIST: Benchmarking Active Tactile Perception
• CLAMP: Crowdsourcing a LArge-scale in-the-wild haptic dataset with an open-source device for Multimodal robot Perception
• ViTaPEs: Visuotactile Position Encodings for Cross-Modal Alignment in Multimodal Transformers
• ControlTac: Force- and Position-Controlled Tactile Data Augmentation with a Single Reference Image
• RA-Touch: Retrieval-Augmented Touch Understanding with Enriched Visual Data
• Adaptive Visuo-Tactile Fusion with Predictive Force Attention for Dexterous Manipulation
• TouchWorld: A Predictive and Reactive Tactile Foundation Model for Dexterous Manipulation

Related

• "Touching to See" and "Seeing to Feel": Robotic Cross-modal SensoryData Generation for Visual-Tactile Perception
• Touch and Go: Learning from Human-Collected Vision and Touch
• Touch2Touch: Cross-Modal Tactile Generation for Object Manipulation
• Binding Touch to Everything: Learning Unified Multimodal Tactile Representations
• Tactile-Augmented Radiance Fields