2020/03/25 by W. X. Wilcke, Xander Wilcke, Wilcke, W. X. +11 · 17 citations
Computer Science · #Advanced Graph Neural Networks #Artificial Intelligence (cs.AI) #Artificial intelligence #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Computer science #End-to-end principle #FOS: Computer and information sciences #Information retrieval #Machine Learning (cs.LG) #Machine learning #Modalities #Modality (human–computer interaction) #Multimodal Machine Learning Applications #Natural language processing #Node (physics) #Relational database #Representation (politics) #Set (abstract data type) #Space (punctuation) #Statistical relational learning #Task (project management) #Theoretical computer science #Topic Modeling #cs.AI #cs.CL #cs.CV #cs.LG
paper · pdf · doi:10.48550/arxiv.2003.12383
published in arXiv (Cornell University) (Cornell University) · Submitted to the 17th International Conference on Principles of Knowledge Representation and Reasoning (2020)
arxiv created 2020/03/25 · openalex publication_date 2020/03/25 · arxiv updated 2020/03/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06
End-to-end multimodal learning on knowledge graphs has been left largely unaddressed. Instead, most end-to-end models such as message passing networks learn solely from the relational information encoded in graphs' structure: raw values, or literals, are either omitted completely or are stripped from their values and treated as regular nodes. In either case we lose potentially relevant information which could have otherwise been exploited by our learning methods. To avoid this, we must treat literals and non-literals as separate cases. We must also address each modality separately and accordingly: numbers, texts, images, geometries, et cetera. We propose a multimodal message passing network which not only learns end-to-end from the structure of graphs, but also from their possibly divers set of multimodal node features. Our model uses dedicated (neural) encoders to naturally learn embeddings for node features belonging to five different types of modalities, including images and geometries, which are projected into a joint representation space together with their relational information. We demonstrate our model on a node classification task, and evaluate the effect that each modality has on the overall performance. Our result supports our hypothesis that including information from multiple modalities can help our models obtain a better overall performance.