2018/05/26 by Volkan Cirik, Taylor Berg-Kirkpatrick, Cirik, Volkan +4 · 2 citations
Computer Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human Pose and Action Recognition #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Neural and Evolutionary Computing (cs.NE) #cs.CL #cs.CV #cs.NE
paper · pdf · doi:10.48550/arxiv.1805.10547
AAAI 2018
arxiv created 2018/05/26 · openalex publication_date 2018/05/26 · arxiv updated 2018/05/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
We introduce GroundNet, a neural network for referring expression recognition -- the task of localizing (or grounding) in an image the object referred to by a natural language expression. Our approach to this task is the first to rely on a syntactic analysis of the input referring expression in order to inform the structure of the computation graph. Given a parse tree for an input expression, we explicitly map the syntactic constituents and relationships present in the tree to a composed graph of neural modules that defines our architecture for performing localization. This syntax-based approach aids localization of both the target object and auxiliary supporting objects mentioned in the expression. As a result, GroundNet is more interpretable than previous methods: we can (1) determine which phrase of the referring expression points to which object in the image and (2) track how the localization of the target object is determined by the network. We study this property empirically by introducing a new set of annotations on the GoogleRef dataset to evaluate localization of supporting objects. Our experiments show that GroundNet achieves state-of-the-art accuracy in identifying supporting objects, while maintaining comparable performance in the localization of target objects.