2022/10/07 by Youngsun Wi, Andy Zeng, Wi, Youngsun +5 · 2 citations
Engineering · Neuroscience · #FOS: Computer and information sciences #Motor Control and Adaptation #Robot Manipulation and Learning #Robotics (cs.RO) #Tactile and Sensory Interactions
paper · pdf · doi:10.48550/arxiv.2210.03701
openalex publication_date 2022/10/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Deformable objects manipulation can benefit from representations that seamlessly integrate vision and touch while handling occlusions. In this work, we present a novel approach for, and real-world demonstration of, multimodal visuo-tactile state-estimation and dynamics prediction for deformable objects. Our approach, VIRDO++, builds on recent progress in multimodal neural implicit representations for deformable object state-estimation [1] via a new formulation for deformation dynamics and a complementary state-estimation algorithm that (i) maintains a belief over deformations, and (ii) enables practical real-world application by removing the need for privileged contact information. In the context of two real-world robotic tasks, we show:(i) high-fidelity cross-modal state-estimation and prediction of deformable objects from partial visuo-tactile feedback, and (ii) generalization to unseen objects and contact formations.