Multi-Modal Graph Neural Network for Joint Reasoning on Vision and Scene Text

2020/03/31 by Difei Gao, Ke Li, Gao, Difei +7 · 4 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2003.13962

openalex publication_date 2020/03/31 · openalex created_date 2020/04/10 · openalex updated_date 2026/07/28

Abstract

Answering questions that require reading texts in an image is challenging for current models. One key difficulty of this task is that rare, polysemous, and ambiguous words frequently appear in images, e.g., names of places, products, and sports teams. To overcome this difficulty, only resorting to pre-trained word embedding models is far from enough. A desired model should utilize the rich information in multiple modalities of the image to help understand the meaning of scene texts, e.g., the prominent text on a bottle is most likely to be the brand. Following this idea, we propose a novel VQA approach, Multi-Modal Graph Neural Network (MM-GNN). It first represents an image as a graph consisting of three sub-graphs, depicting visual, semantic, and numeric modalities respectively. Then, we introduce three aggregators which guide the message passing from one graph to another to utilize the contexts in various modalities, so as to refine the features of nodes. The updated nodes have better features for the downstream question answering module. Experimental evaluations show that our MM-GNN represents the scene texts better and obviously facilitates the performances on two VQA tasks that require reading scene texts.

Citations

• Exploring Models and Data for Image Question Answering
• Long Short-Term Memory
• Learning Conditioned Graph Structures for Interpretable Visual Question\n Answering
• Detecting Functionality-Specific Vulnerabilities via Retrieving Individual Functionality-Equivalent APIs in Open-Source Repositories
• FVQA: Fact-Based Visual Question Answering
• PyTorch: An Imperative Style, High-Performance Deep Learning Library

Cited by

• Expressive and Scalable Quantum Fusion for Multimodal Learning
• Aggregated Structural Representation with Large Language Models for Human-Centric Layout Generation
• PsOCR: Benchmarking Large Multimodal Models for Optical Character Recognition in Low-resource Pashto Language
• CircuitFusion: Multimodal Circuit Representation Learning for Agile Chip Design

Related

• Multi-Modal Reasoning Graph for Scene-Text Based Fine-Grained Image Classification and Retrieval
• Textually Enriched Neural Module Networks for Visual Question Answering
• Bilateral Cross-Modality Graph Matching Attention for Feature Fusion in Visual Question Answering
• VQA-GNN: Reasoning with Multimodal Knowledge via Graph Neural Networks for Visual Question Answering
• SelfGraphVQA: A Self-Supervised Graph Neural Network for Scene-based Question Answering