AttnGrounder: Talking to Cars with Attention

2020/09/11 by Mittal, Vivek · 2 citations
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2009.05684

Abstract

We propose Attention Grounder (AttnGrounder), a single-stage end-to-end trainable model for the task of visual grounding. Visual grounding aims to localize a specific object in an image based on a given natural language text query. Unlike previous methods that use the same text representation for every image region, we use a visual-text attention module that relates each word in the given query with every region in the corresponding image for constructing a region dependent text representation. Furthermore, for improving the localization ability of our model, we use our visual-text attention module to generate an attention mask around the referred object. The attention mask is trained as an auxiliary task using a rectangular mask generated with the provided ground-truth coordinates. We evaluate AttnGrounder on the Talk2Car dataset and show an improvement of 3.26% over the existing methods.

Cited by

• E3AD: An Emotion-Aware Vision-Language-Action Model for Human-Centric End-to-End Autonomous Driving
• Think Before You Drive: World Model-Inspired Multimodal Grounding for Autonomous Vehicles

Related

• Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding
• Commands 4 Autonomous Vehicles (C4AV) Workshop Summary
• Grounding Commands for Autonomous Vehicles via Layer Fusion with Region-specific Dynamic Layer Attention
• Real-Time Referring Expression Comprehension by Single-Stage Grounding Network
• Learning Visual Grounding from Generative Vision and Language Model