Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks

2019/07/27 by Michelle A. Lee, Lee, Michelle A., Yuke Zhu +15 · 27 citations
Computer Science · Engineering · Neuroscience · #Reinforcement Learning in Robotics #Robot Manipulation and Learning #Tactile and Sensory Interactions #cs.LG #cs.RO

paper · pdf · doi:10.48550/arxiv.1907.13098

arXiv admin note: substantial text overlap with arXiv:1810.10191

arxiv created 2019/07/28 · arxiv updated 2019/07/31

Abstract

Contact-rich manipulation tasks in unstructured environments often require both haptic and visual feedback. It is non-trivial to manually design a robot controller that combines these modalities which have very different characteristics. While deep reinforcement learning has shown success in learning control policies for high-dimensional inputs, these algorithms are generally intractable to deploy on real robots due to sample complexity. In this work, we use self-supervision to learn a compact and multimodal representation of our sensory inputs, which can then be used to improve the sample efficiency of our policy learning. Evaluating our method on a peg insertion task, we show that it generalizes over varying geometries, configurations, and clearances, while being robust to external perturbations. We also systematically study different self-supervised learning objectives and representation learning architectures. Results are presented in simulation and on a physical robot.

Citations

• Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks

Cited by

• FeelWorld: Visuo-Tactile World Model for Hierarchical Contact Prediction and Planning
• Explainable Multimodal Regression via Information Decomposition
• UniTacHand: Unified Spatio-Tactile Representation for Human to Robotic Hand Skill Transfer
• Contact-Rich Robotic Assembly in Construction via Diffusion Policy Learning
• Self-Supervised Multisensory Pretraining for Contact-Rich Robot Reinforcement Learning
• Intuitive Programming, Adaptive Task Planning, and Dynamic Role Allocation in Human-Robot Collaboration
• Collaborative Assembly Policy Learning of a Sightless Robot
• End-to-End Dexterous Arm-Hand VLA Policies via Shared Autonomy: VR Teleoperation Augmented by Autonomous Hand VLA Policy for Efficient Data Collection
• Enhancing Tactile-based Reinforcement Learning for Robotic Control
• Representing Data in Robotic Tactile Perception -- A Review
• InfMasking: Unleashing Synergistic Information by Contrastive Multimodal Interactions
• TacWAM: Anchor-Guided World Action Model with Mechanics-Aware Tactile Prediction
• Gentle Object Retraction in Dense Clutter Using Multimodal Force Sensing and Imitation Learning
• Human Centric General Physical Intelligence for Agile Manufacturing Automation
• Tactile Robotics: An Outlook
• OmniVTLA: Vision-Tactile-Language-Action Model with Semantic-Aligned Tactile Sensing
• Few-shot transfer of tool-use skills using human demonstrations with proximity and tactile sensing
• Failure Forecasting Boosts Robustness of Sim2Real Rhythmic Insertion Policies
• Accurate Pose Estimation Using Contact Manifold Sampling for Safe Peg-in-Hole Insertion of Complex Geometries
• ViTacFormer: Learning Cross-Modal Representation for Visuo-Tactile Dexterous Manipulation
• Designing Pin-pression Gripper and Learning its Dexterous Grasping with Online In-hand Adjustment
• Learning the Contact Manifold for Accurate Pose Estimation During Peg-in-Hole Insertion of Complex Geometries
• Demystifying When and Why VLAs Fail in Contact-Rich Tasks and How to Fix Them
• Integrating Model-based Control and RL for Sim2Real Transfer of Tight Insertion Policies
• VTLA: Vision-Tactile-Language-Action Model with Preference Learning for Insertion Manipulation
• ViTa-Zero: Zero-shot Visuotactile Object 6D Pose Estimation
• A General Peg-in-Hole Assembly Policy Based on Domain Randomized Reinforcement Learning

Related

• Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks
• Learning rich touch representations through cross-modal self-supervision
• The Power of the Senses: Generalizable Manipulation from Vision and Touch through Masked Multimodal Learning
• Grasping Deformable Objects via Reinforcement Learning with Cross-Modal Attention to Visuo-Tactile Inputs
• Enhancing Tactile-based Reinforcement Learning for Robotic Control