2020/07/02 by Marcel Hildebrandt, Hang Li, Hildebrandt, Marcel +7 · 14 citations
Computer Science · Mathematics · #Advanced Image and Video Retrieval Techniques #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Multimodal Machine Learning Applications #cs.CV #cs.LG #stat.ML
paper · pdf · doi:10.48550/arxiv.2007.01072
ICML Workshop Graph Representation Learning and Beyond (GRL+)
arxiv created 2020/07/02 · openalex publication_date 2020/07/02 · arxiv updated 2020/07/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Visual question answering is concerned with answering free-form questions about an image. Since it requires a deep linguistic understanding of the question and the ability to associate it with various objects that are present in the image, it is an ambitious task and requires techniques from both computer vision and natural language processing. We propose a novel method that approaches the task by performing context-driven, sequential reasoning based on the objects and their semantic and spatial relationships present in the scene. As a first step, we derive a scene graph which describes the objects in the image, as well as their attributes and their mutual relationships. A reinforcement agent then learns to autonomously navigate over the extracted scene graph to generate paths, which are then the basis for deriving answers. We conduct a first experimental study on the challenging GQA dataset with manually curated scene graphs, where our method almost reaches the level of human performance.