Navigation Instruction Generation with BEV Perception and Large Language Models

2024/07/21 by Sheng Fan, Fan, Sheng, Rui Liu +5 · 9 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech and dialogue systems

paper · pdf · doi:10.48550/arxiv.2407.15087

openalex publication_date 2024/07/21 · openalex created_date 2025/01/05 · openalex updated_date 2026/07/28

Abstract

Navigation instruction generation, which requires embodied agents to describe the navigation routes, has been of great interest in robotics and human-computer interaction. Existing studies directly map the sequence of 2D perspective observations to route descriptions. Though straightforward, they overlook the geometric information and object semantics of the 3D environment. To address these challenges, we propose BEVInstructor, which incorporates Bird's Eye View (BEV) features into Multi-Modal Large Language Models (MLLMs) for instruction generation. Specifically, BEVInstructor constructs a PerspectiveBEVVisual Encoder for the comprehension of 3D environments through fusing BEV and perspective features. To leverage the powerful language capabilities of MLLMs, the fused representations are used as visual prompts for MLLMs, and perspective-BEV prompt tuning is proposed for parameter-efficient updating. Based on the perspective-BEV prompts, BEVInstructor further adopts an instance-guided iterative refinement pipeline, which improves the instructions in a progressive manner. BEVInstructor achieves impressive performance across diverse datasets (i.e., R2R, REVERIE, and UrbanWalk).

Cited by

• CVP: Central-Peripheral Vision-Inspired Multimodal Model for Spatial Reasoning
• NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigation
• Strategic Fusion of Vision Language Models: Shapley-Credited Context-Aware Dawid-Skene for Multi-Label Tasks in Autonomous Driving
• Text-Scene: A Scene-to-Language Parsing Framework for 3D Scene Understanding
• GoViG: Goal-Conditioned Visual Navigation Instruction Generation
• Generating Vision-Language Navigation Instructions Incorporated Fine-Grained Alignment Annotations
• Silence is Golden: Leveraging Adversarial Examples to Nullify Audio Control in LDM-based Talking-Head Generation
• SignBot: Learning Human-to-Humanoid Sign Language Interaction
• Cross from Left to Right Brain: Adaptive Text Dreamer for Vision-and-Language Navigation

Related

• Following Route Instructions using Large Vision-Language Models: A Comparison between Low-level and Panoramic Action Spaces
• NavA³: Understanding Any Instruction, Navigating Anywhere, Finding Anything
• BEVDriver: Leveraging BEV Maps in LLMs for Robust Closed-Loop Driving
• Bird's-Eye-View Scene Graph for Vision-Language Navigation
• OpenNav: Open-World Navigation with Multimodal Large Language Models