IQA: Visual Question Answering in Interactive Environments

2017/12/09 by Daniel Gordon, Aniruddha Kembhavi, Gordon, Daniel +9 · 17 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #cs.CV

paper · pdf · doi:10.48550/arxiv.1712.03316

Published in CVPR 2018

arxiv created 2018/09/06 · arxiv updated 2018/09/07

Abstract

We introduce Interactive Question Answering (IQA), the task of answering questions that require an autonomous agent to interact with a dynamic visual environment. IQA presents the agent with a scene and a question, like: "Are there any apples in the fridge?" The agent must navigate around the scene, acquire visual understanding of scene elements, interact with objects (e.g. open refrigerators) and plan for a series of actions conditioned on the question. Popular reinforcement learning approaches with a single controller perform poorly on IQA owing to the large and diverse state space. We propose the Hierarchical Interactive Memory Network (HIMN), consisting of a factorized set of controllers, allowing the system to operate at multiple levels of temporal abstraction. To evaluate HIMN, we introduce IQUAD V1, a new dataset built upon AI2-THOR, a simulated photo-realistic environment of configurable indoor scenes with interactive objects (code and dataset available at https://github.com/danielgordon10/thor-iqa-cvpr-2018). IQUAD V1 has 75,000 questions, each paired with a unique scene configuration. Our experiments show that our proposed model outperforms popular single controller based methods on IQUAD V1. For sample questions and results, please view our video: https://youtu.be/pXd3C-1jr98

Cited by

• When Robots Should Say "I Don't Know": Benchmarking Abstention in Embodied Question Answering
• Vision to Geometry: 3D Spatial Memory for Sequential Embodied MLLM Reasoning and Exploration
• ReEXplore: Improving MLLMs for Embodied Exploration with Contextualized Retrospective Experience Replay
• Searching in Space and Time: Unified Memory-Action Loops for Open-World Object Retrieval
• Multi-Step Reasoning for Embodied Question Answering via Tool Augmentation
• Situat3DChange: Situated 3D Change Understanding Dataset for Multimodal Large Language Model
• Agentic Services Computing
• Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Survey
• mKG-RAG: Multimodal Knowledge Graph-Enhanced RAG for Visual Question Answering
• PilotRL: Training Language Model Agents via Global Planning-Guided Progressive Reinforcement Learning
• Enhancing Spatial Reasoning through Visual and Textual Thinking
• Enter the Mind Palace: Reasoning and Planning for Long-term Active Embodied Question Answering
• LangScene-X: Reconstruct Generalizable 3D Language-Embedded Scenes with TriMap Video Diffusion
• GroundFlow: A Plug-in Module for Temporal Reasoning on 3D Point Cloud Sequential Grounding
• EQA-RM: A Generative Embodied Reward Model with Test-time Scaling
• IndustryEQA: Pushing the Frontiers of Embodied Question Answering in Industrial Scenarios
• Memory-Centric Embodied Question Answering
• AerialVG: A Challenging Benchmark for Aerial Visual Grounding by Exploring Positional Relations

Related

• Visual Curiosity: Learning to Ask Questions to Learn Visual Recognition
• Embodied Question Answering
• IQ-VQA: Intelligent Visual Question Answering
• Can Open Domain Question Answering Systems Answer Visual Knowledge Questions?
• Hyper-dimensional computing for a visual question-answering system that is trainable end-to-end