Visuo-Tactile based Predictive Cross Modal Perception for Object Exploration in Robotics

2024/05/21 by Anirvan Dutta, Etienne Burdet, Dutta, Anirvan +3
Computer Science · Neuroscience · #FOS: Computer and information sciences #Interactive and Immersive Displays #Robotics (cs.RO) #Tactile and Sensory Interactions #Visual Attention and Saliency Detection

paper · pdf · doi:10.48550/arxiv.2405.12634

openalex publication_date 2024/05/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

Autonomously exploring the unknown physical properties of novel objects such as stiffness, mass, center of mass, friction coefficient, and shape is crucial for autonomous robotic systems operating continuously in unstructured environments. We introduce a novel visuo-tactile based predictive cross-modal perception framework where initial visual observations (shape) aid in obtaining an initial prior over the object properties (mass). The initial prior improves the efficiency of the object property estimation, which is autonomously inferred via interactive non-prehensile pushing and using a dual filtering approach. The inferred properties are then used to enhance the predictive capability of the cross-modal function efficiently by using a human-inspired `surprise' formulation. We evaluated our proposed framework in the real-robotic scenario, demonstrating superior performance.

Related

• Predictive Visuo-Tactile Interactive Perception Framework for Object Properties Inference
• Robotic Perception with a Large Tactile-Vision-Language Model for Physical Property Inference
• Push to know! -- Visuo-Tactile based Active Object Parameter Inference with Dual Differentiable Filtering
• Learning Intuitive Physics with Multimodal Generative Models
• Deep Learning for Tactile Understanding From Visual and Haptic Data