Leveraging Language for Accelerated Learning of Tool Manipulation

2022/06/27 by Allen Z. Ren, Ren, Allen Z., Bharat Govil +7 · 1 citation
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Robotics (cs.RO) #Software Engineering Research

paper · pdf · doi:10.48550/arxiv.2206.13074

openalex publication_date 2022/06/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Robust and generalized tool manipulation requires an understanding of the properties and affordances of different tools. We investigate whether linguistic information about a tool (e.g., its geometry, common uses) can help control policies adapt faster to new tools for a given task. We obtain diverse descriptions of various tools in natural language and use pre-trained language models to generate their feature representations. We then perform language-conditioned meta-learning to learn policies that can efficiently adapt to new tools given their corresponding text descriptions. Our results demonstrate that combining linguistic information and meta-learning significantly accelerates tool learning in several manipulation tasks including pushing, lifting, sweeping, and hammering.

Cited by

• CLASP: General-Purpose Clothes Manipulation with Semantic Keypoints

Related

• ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases
• Flexible Tool Selection through Low-dimensional Attribute Alignment of Vision and Language
• MetaTool: Facilitating Large Language Models to Master Tools with Meta-task Augmentation
• Learning Generalizable Tool-use Skills through Trajectory Generation
• ToolNet: Connecting Large Language Models with Massive Tools via Tool Graph