General In-Hand Object Rotation with Vision and Touch

2023/09/18 by Haozhi Qi, Brent Yi, Qi, Haozhi +11 · 30 citations
Neuroscience · Computer Science · #Tactile and Sensory Interactions #Interactive and Immersive Displays #Hand Gesture Recognition Systems

paper · pdf · doi:10.48550/arxiv.2309.09979

Abstract

We introduce RotateIt, a system that enables fingertip-based object rotation along multiple axes by leveraging multimodal sensory inputs. Our system is trained in simulation, where it has access to ground-truth object shapes and physical properties. Then we distill it to operate on realistic yet noisy simulated visuotactile and proprioceptive sensory inputs. These multimodal inputs are fused via a visuotactile transformer, enabling online inference of object shapes and physical properties during deployment. We show significant performance improvements over prior methods and the importance of visual and tactile sensing.

Cited by

• Embodied Robot Manipulation in the Era of Foundation Models: Planning and Learning Perspectives
• World Models Can Leverage Human Videos for Dexterous Manipulation
• Simultaneous Tactile-Visual Perception for Learning Multimodal Robot Manipulation
• OmniDexVLG: Learning Dexterous Grasp Generation from Vision Language Model-Guided Grasp Semantics, Taxonomy and Functional Affordance
• Learning Dexterous Manipulation Skills from Imperfect Simulations
• Is Image-based Object Pose Estimation Ready to Support Grasping?
• Toward Artificial Palpation: Representation Learning of Touch on Soft Bodies
• SPIDER: Scalable Physics-Informed Dexterous Retargeting
• SpikeATac: A Multimodal Tactile Finger with Taxelized Dynamic Sensing for Dexterous Manipulation
• Enhancing Tactile-based Reinforcement Learning for Robotic Control
• Towards Automated Chicken Deboning via Learning-based Dynamically-Adaptive 6-DoF Multi-Material Cutting
• RAPID Hand Prototype: Design of an Affordable, Fully-Actuated Biomimetic Hand for Dexterous Teleoperation
• VT-Refine: Learning Bimanual Assembly with Visuo-Tactile Feedback via Simulation Fine-Tuning
• DexNDM: Closing the Reality Gap for Dexterous In-Hand Rotation via Joint-Wise Neural Dynamics Model
• ISyHand: A Dexterous Multi-finger Robot Hand with an Articulated Palm
• Multi-Modal Manipulation via Multi-Modal Policy Consensus
• In-Hand Manipulation of Articulated Tools with Dexterous Robot Hands with Sim-to-Real Transfer
• UniCross: Unified Cross-Skill Dexterous Manipulation Synthesis
• DexSkin: High-Coverage Conformable Robotic Skin for Learning Contact-Rich Manipulation
• Residual Off-Policy RL for Finetuning Behavior Cloning Policies
• Text2Touch: Tactile In-Hand Manipulation with LLM-Designed Reward Functions
• DEXOP: A Device for Robotic Transfer of Dexterous Human Manipulation
• Classification of Vision-Based Tactile Sensors: A Review
• Poke and Strike: Learning Task-Informed Exploration Policies
• Deep Sensorimotor Control by Imitating Predictive Models of Human Motion
• FBI: Learning Dexterous In-hand Manipulation with Dynamic Visuotactile Shortcut Policy
• Tactile Robotics: An Outlook
• DexReMoE:In-hand Reorientation of General Object via Mixtures of Experts
• Learning Pivoting Manipulation with Force and Vision Feedback Using Optimization-based Demonstrations
• Flow Matching Policy Gradients

Related

• Rotating without Seeing: Towards In-hand Dexterity through Touch
• Rotating Objects via In-Hand Pivoting using Vision, Force and Touch
• AnyRotate: Gravity-Invariant In-Hand Object Rotation with Sim-to-Real Touch
• Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensing
• Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation