2022/01/25 by Peixi Xiong, Quanzeng You, Xiong, Peixi +7
Computer Science · #Adjacency list #Advanced Image and Video Retrieval Techniques #Artificial intelligence #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Graph #Information retrieval #Interpretability #Machine learning #Modalities #Modality (human–computer interaction) #Multimodal Machine Learning Applications #Natural language processing #Question answering #Representation (politics) #Theoretical computer science #Visual reasoning #cs.CV
paper · pdf · doi:10.48550/arxiv.2201.10654
arxiv created 2022/01/25 · openalex publication_date 2022/01/25 · arxiv updated 2022/01/27 · openalex created_date 2022/04/03 · openalex updated_date 2026/07/28
Visual Question Answering (VQA) attracts much attention from both industry and academia. As a multi-modality task, it is challenging since it requires not only visual and textual understanding, but also the ability to align cross-modality representations. Previous approaches extensively employ entity-level alignments, such as the correlations between the visual regions and their semantic labels, or the interactions across question words and object features. These attempts aim to improve the cross-modality representations, while ignoring their internal relations. Instead, we propose to apply structured alignments, which work with graph representation of visual and textual content, aiming to capture the deep connections between the visual and textual modalities. Nevertheless, it is nontrivial to represent and integrate graphs for structured alignments. In this work, we attempt to solve this issue by first converting different modality entities into sequential nodes and the adjacency graph, then incorporating them for structured alignments. As demonstrated in our experimental results, such a structured alignment improves reasoning performance. In addition, our model also exhibits better interpretability for each generated answer. The proposed model, without any pretraining, outperforms the state-of-the-art methods on GQA dataset, and beats the non-pretrained state-of-the-art methods on VQA-v2 dataset.