SparseDFF: Sparse-View Feature Distillation for One-Shot Dexterous Manipulation

2023/10/25 by Qianxu Wang, Haotong Zhang, Wang, Qianxu +11 · 6 citations
Computer Science · Engineering · #Domain Adaptation and Few-Shot Learning #Robot Manipulation and Learning #Image Processing Techniques and Applications

paper · pdf · doi:10.48550/arxiv.2310.16838

Abstract

Humans demonstrate remarkable skill in transferring manipulation abilities across objects of varying shapes, poses, and appearances, a capability rooted in their understanding of semantic correspondences between different instances. To equip robots with a similar high-level comprehension, we present SparseDFF, a novel DFF for 3D scenes utilizing large 2D vision models to extract semantic features from sparse RGBD images, a domain where research is limited despite its relevance to many tasks with fixed-camera setups. SparseDFF generates view-consistent 3D DFFs, enabling efficient one-shot learning of dexterous manipulations by mapping image features to a 3D point cloud. Central to SparseDFF is a feature refinement network, optimized with a contrastive loss between views and a point-pruning mechanism for feature continuity. This facilitates the minimization of feature discrepancies w.r.t. end-effector parameters, bridging demonstrations and target manipulations. Validated in real-world scenarios with a dexterous hand, SparseDFF proves effective in manipulating both rigid and deformable objects, demonstrating significant generalization capabilities across object and scene variations.

Cited by

• Actron3D: Learning Actionable Neural Functions from Videos for Transferable Robotic Manipulation
• VLA-R1: Enhancing Reasoning in Vision-Language-Action Models
• Parse-Augment-Distill: Learning Generalizable Bimanual Visuomotor Policies from Single Human Video
• Point2Act: Efficient 3D Distillation of Multimodal LLMs for Zero-Shot Context-Aware Grasping
• A Survey: Learning Embodied Intelligence from Physical Simulators and World Models
• ManipLVM-R1: Reinforcement Learning for Reasoning in Embodied Manipulation with Large Vision-Language Models

Related

• LensDFF: Language-enhanced Sparse Feature Distillation for Efficient Few-Shot Dexterous Manipulation
• SparseGrasp: Robotic Grasping via 3D Semantic Gaussian Splatting from Sparse Multi-View RGB Images
• Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation
• Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis
• DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Features