New Ideas and Trends in Deep Multimodal Content Understanding: A Review

2020/10/16 by Wei Chen, Chen, Wei, Weiping Wang +5
Computer Science · #Advanced Image and Video Retrieval Techniques #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2010.08189

openalex publication_date 2020/10/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The focus of this survey is on the analysis of two modalities of multimodal deep learning: image and text. Unlike classic reviews of deep learning where monomodal image classifiers such as VGG, ResNet and Inception module are central topics, this paper will examine recent multimodal deep models and structures, including auto-encoders, generative adversarial nets and their variants. These models go beyond the simple image classifiers in which they can do uni-directional (e.g. image captioning, image generation) and bi-directional (e.g. cross-modal retrieval, visual question answering) multimodal tasks. Besides, we analyze two aspects of the challenge in terms of better content understanding in deep multimodal applications. We then introduce current ideas and trends in deep multimodal feature learning, such as feature embedding approaches and objective function design, which are crucial in overcoming the aforementioned challenges. Finally, we include several promising directions for future research.

Citations

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Generative Adversarial Text to Image Synthesis
• Semi-Supervised Classification with Graph Convolutional Networks
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Deep Multimodal Learning: A Survey on Recent Advances and Trends
• An Introduction to Image Synthesis with Generative Adversarial Nets
• Self-Attention Generative Adversarial Networks
• Auto-Encoding Scene Graphs for Image Captioning
• Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answering
• Graph Neural Networks: A Review of Methods and Applications
• Relation-Aware Graph Attention Network for Visual Question Answering
• Learning to Guide Decoding for Image Captioning
• Exploring Human-like Attention Supervision in Visual Question Answering
• A Comprehensive Survey on Graph Neural Networks

Related

• Deep Multimodal Learning: A Survey on Recent Advances and Trends
• Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications
• Multimodal Deep Learning
• Learning Multimodal Data Augmentation in Feature Space
• Show, Translate and Tell