On Evaluation of Embodied Navigation Agents

2018/07/18 by Peter Anderson, Angel Chang, Anderson, Peter +21 · 139 citations
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Robotic Path Planning Algorithms #Robotics (cs.RO) #Robotics and Sensor-Based Localization #cs.AI #cs.CV #cs.LG #cs.RO

paper · pdf · doi:10.48550/arxiv.1807.06757

Report of a working group on empirical methodology in navigation research. Authors are listed in alphabetical order

arxiv created 2018/07/18 · openalex publication_date 2018/07/18 · arxiv updated 2018/07/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Skillful mobile operation in three-dimensional environments is a primary topic of study in Artificial Intelligence. The past two years have seen a surge of creative work on navigation. This creative output has produced a plethora of sometimes incompatible task definitions and evaluation protocols. To coordinate ongoing and future research in this area, we have convened a working group to study empirical methodology in navigation research. The present document summarizes the consensus recommendations of this working group. We discuss different problem statements and the role of generalization, present evaluation measures, and provide standard scenarios that can be used for benchmarking.

Citations

• Matterport3D: Learning from RGB-D Data in Indoor Environments
• HoME: a Household Multimodal Environment
• AI2-THOR: An Interactive 3D Environment for Visual AI
• Building Generalizable Agents with a Realistic and Rich 3D Environment

Cited by

• VL-LN Bench: Towards Long-horizon Goal-oriented Navigation with Active Dialogs
• SONG: A Photorealistic 3D Gaussian Simulation Platform for Benchmarking Social Navigation
• Room-Mediated Co-occurrence for Zero-Shot Object-Centric Semantic Navigation via Frontier Scoring
• ImagineNav++: Prompting Vision-Language Models as Embodied Navigator through Scene Imagination
• AstraNav-World: World Model for Foresight Control and Consistency
• Schrödinger's Navigator: Imagining an Ensemble of Futures for Zero-Shot Object Navigation
• Learning Category-level Last-meter Navigation from RGB Demonstrations of a Single-instance
• User-Feedback-Driven Adaptation for Vision-and-Language Navigation
• Vision to Geometry: 3D Spatial Memory for Sequential Embodied MLLM Reasoning and Exploration
• FOM-Nav: Frontier-Object Maps for Object Goal Navigation
• Prune-Then-Plan: Step-Level Calibration for Stable Frontier Exploration in Embodied Question Answering
• Navigating the Wild: Pareto-Optimal Visual Decision-Making in Image Space
• UrbanVLA: A Vision-Language-Action Model for Urban Micromobility
• HyPerNav: Hybrid Perception for Object-Oriented Navigation in Unknown Environment
• Transferable Representation Learning in Vision-and-Language Navigation
• No RL, No Simulation: Learning to Navigate without Navigating
• NeSyPr: Neurosymbolic Proceduralization For Efficient Embodied Reasoning
• Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-training
• Embodied Navigation with Auxiliary Task of Action Description Prediction
• SSCNav: Confidence-Aware Semantic Scene Completion for Visual Semantic Navigation
• Auxiliary Tasks Speed Up Learning PointGoal Navigation
• Audio-Guided Visual Perception for Audio-Visual Navigation
• Towards Safe Maneuvering of Double-Ackermann-Steering Robots with a Soft Actor-Critic Framework
• What Matters in RL-Based Methods for Object-Goal Navigation? An Empirical Study and A Unified Framework
• Scene Memory Transformer for Embodied Agents in Long-Horizon Tasks
• Deep Learning for Embodied Vision Navigation: A Survey
• USIM and U0: A Vision-Language-Action Dataset and Model for General Underwater Robots
• Dream to Recall: Imagination-Guided Experience Retrieval for Memory-Persistent Vision-and-Language Navigation
• SegMASt3R: Geometry Grounded Segment Matching
• CORE-3D: Context-aware Open-vocabulary Retrieval by Embeddings in 3D
• CE-Nav: Flow-Guided Reinforcement Refinement for Cross-Embodiment Local Navigation
• HELIOS: Hierarchical Exploration for Language-grounded Interaction in Open Scenes
• Towards Target-Driven Visual Navigation in Indoor Scenes via Generative Imitation Learning
• Where Did I Leave My Glasses? Open-Vocabulary Semantic Exploration in Real-World Semi-Static Environments
• Learning Social Robot Navigation By Sensing Human Legs
• Offline Preference-Based Trajectory Evaluation
• A scalar per patch from pre-trained ViTs enables fast moving navigation in the real world
• Analyzing Visual Representations in Embodied Navigation Tasks
• Learning to Map for Active Semantic Goal Navigation
• Sight Over Site: Perception-Aware Reinforcement Learning for Efficient Robotic Inspection
• FiLM-Nav: Efficient and Generalizable Navigation via VLM Fine-tuning
• From reactive to cognitive: brain-inspired spatial intelligence for embodied agents
• Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout
• Synthetic vs. Real Training Data for Visual Navigation
• GC-VLN: Instruction as Graph Constraints for Training-free Vision-and-Language Navigation
• ObjectReact: Learning Object-Relative Control for Visual Navigation
• Learning hierarchical relationships for object-goal navigation
• A Recurrent Vision-and-Language BERT for Navigation
• SoundSpaces: Audio-Visual Navigation in 3D Environments
• VALAN: Vision and Language Agent Navigation
• A Survey of Embodied AI: From Simulators to Research Tasks
• Sim-Real Joint Reinforcement Transfer for 3D Indoor Navigation
• TopoNav: Topological Graphs as a Key Enabler for Advanced Object Navigation
• LanguageRefer: Spatial-Language Model for 3D Visual Grounding
• Deep Visual MPC-Policy Learning for Navigation
• Active Imitation Learning from Multiple Non-Deterministic Teachers: Formulation, Challenges, and Algorithms
• SIGN: Safety-Aware Image-Goal Navigation for Autonomous Drones via Reinforcement Learning
• Point Cloud Based Reinforcement Learning for Sim-to-Real and Partial Observability in Visual Navigation
• MAG-Nav: Language-Driven Object Navigation Leveraging Memory-Reserved Active Grounding
• Open Scene Graphs for Open-World Object-Goal Navigation
• Following Route Instructions using Large Vision-Language Models: A Comparison between Low-level and Panoramic Action Spaces
• VPN: Visual Prompt Navigation
• UAV-ON: A Benchmark for Open-World Object Goal Navigation with Aerial Agents
• Successor Features for Transfer in Alternating Markov Games
• When Engineering Outruns Intelligence: Rethinking Instruction-Guided Navigation
• OctoNav: Towards Generalist Embodied Navigation
• Learning to Navigate in Indoor Environments: from Memorizing to Reasoning
• Embodied Vision-and-Language Navigation with Dynamic Convolutional Filters
• X-Nav: Learning End-to-End Cross-Embodiment Navigation for Mobile Robots
• Semantic Visual Navigation by Watching YouTube Videos
• NeoNav: Improving the Generalization of Visual Navigation via Generating Next Expected Observations
• Efficient Robotic Object Search via HIEM: Hierarchical Policy Learning with Intrinsic-Extrinsic Modeling
• Enter the Mind Palace: Reasoning and Planning for Long-term Active Embodied Question Answering
• An Inductive Bias for Distances: Neural Nets that Respect the Triangle\n Inequality
• Learning to Visually Navigate in Photorealistic Environments Without any\n Supervision
• CogDDN: A Cognitive Demand-Driven Navigation with Decision Optimization and Dual-Process Thinking
• Cross-Lingual Vision-Language Navigation
• Towards real-world navigation with deep differentiable planners
• Language and Visual Entity Relationship Graph for Agent Navigation
• Topological Planning with Transformers for Vision-and-Language Navigation
• Multi-modal Discriminative Model for Vision-and-Language Navigation
• Object-and-Action Aware Model for Visual Language Navigation
• Soft Expert Reward Learning for Vision-and-Language Navigation
• Generative Language-Grounded Policy in Vision-and-Language Navigation with Bayes' Rule
• OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understanding
• The Replica Dataset: A Digital Replica of Indoor Spaces
• View Invariant Learning for Vision-Language Navigation in Continuous Environments
• Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information
• Learning Exploration Policies for Navigation
• RoboEval: Where Robotic Manipulation Meets Structured and Scalable Evaluation
• NavMorph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environments
• Using RGB Image as Visual Input for Mapless Robot Navigation
• Reinforcement Learning with Physics-Informed Symbolic Program Priors for Zero-Shot Wireless Indoor Navigation
• Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation
• Hierarchical Scoring with 3D Gaussian Splatting for Instance Image-Goal Navigation
• VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuning
• History-Augmented Vision-Language Models for Frontier-Based Zero-Shot Object Navigation
• Efficient Navigation Among Movable Obstacles using a Mobile Manipulator via Hierarchical Policy Learning
• Weakly-supervised VLM-guided Partial Contrastive Learning for Visual Language Navigation
• Pushing it out of the Way: Interactive Visual Navigation
• What Should I Do Now? Marrying Reinforcement Learning and Symbolic Planning
• Self-Monitoring Navigation Agent via Auxiliary Progress Estimation
• ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation
• Multimodal Spatial Language Maps for Robot Navigation and Manipulation
• GENMANIP: LLM-driven Simulation for Generalizable Instruction-Following Manipulation
• Habitat 2.0: Training Home Assistants to Rearrange their Habitat
• To Learn or Not to Learn: Analyzing the Role of Learning for Navigation in Virtual Environments
• LogisticsVLN: Vision-Language Navigation For Low-Altitude Terminal Delivery Based on Agentic UAVs
• FlySearch: Exploring how vision-language models explore
• WoMAP: World Models For Embodied Open-Vocabulary Object Localization
• GraspVLA: a Grasping Foundation Model Pre-trained on Billion-scale Synthetic Action Data
• Multimodal Attention Networks for Low-Level Vision-and-Language Navigation
• Cross from Left to Right Brain: Adaptive Text Dreamer for Vision-and-Language Navigation
• Chasing Ghosts: Instruction Following as Bayesian State Tracking
• SD-OVON: A Semantics-aware Dataset and Benchmark Generation Pipeline for Open-Vocabulary Object Navigation in Dynamic Scenes
• Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling
• CoNav: Collaborative Cross-Modal Reasoning for Embodied Navigation
• The Regretful Agent: Heuristic-Aided Navigation through Progress Estimation
• Place Cells as Multi-Scale Position Embeddings: Random Walk Transition Kernels for Path Planning
• Multi-Agent Embodied Visual Semantic Navigation with Scene Prior Knowledge
• Lessons from reinforcement learning for biological representations of space
• Let Humanoids Hike! Integrative Skill Development on Complex Trails
• Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigation
• Path-conditioned Reinforcement Learning-based Local Planning for Long-Range Navigation
• Learning Semantic Priorities for Autonomous Target Search
• Towards Autonomous Micromobility through Scalable Urban Simulation
• Agentic Abstention: Do Agents Know When to Stop Instead of Act?
• SASRA: Semantically-aware Spatio-temporal Reasoning Agent for Vision-and-Language Navigation in Continuous Environments
• A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AI
• Bayesian Relational Memory for Semantic Visual Navigation
• Sim-to-Real Transfer for Vision-and-Language Navigation
• Auxiliary Tasks and Exploration Enable ObjectNav
• RSRNav: Reasoning Spatial Relationship for Image-Goal Navigation
• Rethinking the Spatial Route Prior in Vision-and-Language Navigation
• ForesightNav: Learning Scene Imagination for Efficient Exploration
• Multimodal Perception for Goal-oriented Navigation: A Survey
• Learning to Set Waypoints for Audio-Visual Navigation
• RoCUS: Robot Controller Understanding via Sampling
• CL-CoTNav: Closed-Loop Hierarchical Chain-of-Thought for Zero-Shot Object-Goal Navigation with Vision-Language Models

Related

• NavSpace: How Navigation Agents Follow Spatial Intelligence Instructions
• Sensing, Social, and Motion Intelligence in Embodied Navigation: A Comprehensive Survey
• Safety of Embodied Navigation: A Survey
• ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects
• Towards Learning a Generalist Model for Embodied Navigation