2022/11/26 by Priya Sundaresan, Sundaresan, Priya, Suneel Belkhale +3 · 4 citations
Computer Science · Engineering · Neuroscience · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Hand Gesture Recognition Systems #Machine Learning (cs.LG) #Robot Manipulation and Learning #Robotics (cs.RO) #Tactile and Sensory Interactions
paper · pdf · doi:10.48550/arxiv.2211.14648
openalex publication_date 2022/11/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Acquiring food items with a fork poses an immense challenge to a robot-assisted feeding system, due to the wide range of material properties and visual appearances present across food groups. Deformable foods necessitate different skewering strategies than firm ones, but inferring such characteristics for several previously unseen items on a plate remains nontrivial. Our key insight is to leverage visual and haptic observations during interaction with an item to rapidly and reactively plan skewering motions. We learn a generalizable, multimodal representation for a food item from raw sensory inputs which informs the optimal skewering strategy. Given this representation, we propose a zero-shot framework to sense visuo-haptic properties of a previously unseen item and reactively skewer it, all within a single interaction. Real-robot experiments with foods of varying levels of visual and textural diversity demonstrate that our multimodal policy outperforms baselines which do not exploit both visual and haptic cues or do not reactively plan. Across 6 plates of different food items, our proposed framework achieves 71% success over 69 skewering attempts total. Supplementary material, datasets, code, and videos are available on our website: https://sites.google.com/view/hapticvisualnet-corl22/home