Improving Long-Horizon Imitation Through Instruction Prediction

2023/06/21 by Joey Hejna, Pieter Abbeel, Hejna, Joey +3 · 4 citations
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Algorithms #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2306.12554

openalex publication_date 2023/06/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Complex, long-horizon planning and its combinatorial nature pose steep challenges for learning-based agents. Difficulties in such settings are exacerbated in low data regimes where over-fitting stifles generalization and compounding errors hurt accuracy. In this work, we explore the use of an often unused source of auxiliary supervision: language. Inspired by recent advances in transformer-based models, we train agents with an instruction prediction loss that encourages learning temporally extended representations that operate at a high level of abstraction. Concretely, we demonstrate that instruction modeling significantly improves performance in planning environments when training with a limited number of demonstrations on the BabyAI and Crafter benchmarks. In further analysis we find that instruction modeling is most important for tasks that require complex reasoning, while understandably offering smaller gains in environments that require simple plans. More details and code can be found at https://github.com/jhejna/instruction-prediction.

Cited by

• Imitation Learning in the Deep Learning Era: A Novel Taxonomy and Recent Advances
• Embodied Navigation with Auxiliary Task of Action Description Prediction
• Multi-Agent Taskforce Collaboration: Self-Correction of Compounding Errors in Long-Form Literature Review Generation
• Wavelet Policy: Lifting Scheme for Policy Learning in Long-Horizon Tasks

Related

• Learning Instruction-Following Policies through Open-Ended Instruction Relabeling with Large Language Models
• Training with Pseudo-Code for Instruction Following
• Do LLMs "know" internally when they follow instructions?
• Few-shot Subgoal Planning with Language Models
• Learning and Leveraging Verifiers to Improve Planning Capabilities of Pre-trained Language Models