2023/08/01 by Yuhao Lu, Lu, Yuhao, Yixuan Fan +9 · 8 citations
Computer Science · Engineering · #FOS: Computer and information sciences #Human Pose and Action Recognition #Multimodal Machine Learning Applications #Robot Manipulation and Learning #Robotics (cs.RO)
paper · pdf · doi:10.48550/arxiv.2308.00640
openalex publication_date 2023/08/01 · openalex created_date 2023/08/18 · openalex updated_date 2026/07/28
Robotic grasping faces new challenges in human-robot-interaction scenarios. We consider the task that the robot grasps a target object designated by human's language directives. The robot not only needs to locate a target based on vision-and-language information, but also needs to predict the reasonable grasp pose candidate at various views and postures. In this work, we propose a novel interactive grasp policy, named Visual-Lingual-Grasp (VL-Grasp), to grasp the target specified by human language. First, we build a new challenging visual grounding dataset to provide functional training data for robotic interactive perception in indoor environments. Second, we propose a 6-Dof interactive grasp policy combined with visual grounding and 6-Dof grasp pose detection to extend the universality of interactive grasping. Third, we design a grasp pose filter module to enhance the performance of the policy. Experiments demonstrate the effectiveness and extendibility of the VL-Grasp in real world. The VL-Grasp achieves a success rate of 72.5% in different indoor scenes. The code and dataset is available at https://github.com/luyh20/VL-Grasp.