Deep Multimodal Learning: A Survey on Recent Advances and Trends

2017/11/01 by Dhanesh Ramachandram, Graham W. Taylor · 21 citations
Computer Science · #Multimodal Machine Learning Applications #Domain Adaptation and Few-Shot Learning #Human Pose and Action Recognition

paper · doi:10.1109/msp.2017.2738401

Abstract

The success of deep learning has been a catalyst to solving increasingly complex machine-learning problems, which often involve multiple data modalities. We review recent advances in deep multimodal learning and highlight the state-of the art, as well as gaps and challenges in this active research field. We first classify deep multimodal learning architectures and then discuss methods to fuse learned multimodal representations in deep-learning architectures. We highlight two areas of research-regularization strategies and methods that learn or optimize multimodal fusion structures-as exciting areas for future work.

Citations

• Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects
• Exploring Models and Data for Image Question Answering
• Auto-Encoding Variational Bayes
• Learning and development in neural networks: the importance of starting small
• Taking the Human Out of the Loop: A Review of Bayesian Optimization
• Neural Architecture Search with Reinforcement Learning
• 1 year, 1000 km: The Oxford RobotCar dataset
• Training and Evaluating Multimodal Word Embeddings with Large-scale Web Annotated Images
• Structure Optimization for Deep Multimodal Fusion Networks using Graph-Induced Kernels

Cited by

• How to Leverage Multimodal EHR Data for Better Medical Predictions?
• <scp>MRI‐Based</scp> Multiple Instance Convolutional Neural Network for Increased Accuracy in the Differentiation of Borderline and Malignant Epithelial Ovarian Tumors
• Spectral Thresholds in Correlated Spiked Models and Fundamental Limits of Partial Least Squares
• A Multimodal Approach to SME Credit Scoring Integrating Transaction and Ownership Networks
• A Multisensory Learning Architecture for Rotation-invariant Object Recognition
• MIDAS: Misalignment-based Data Augmentation Strategy for Imbalanced Multimodal Learning
• Social media data for conservation science: A methodological overview
• Facing &amp; mitigating common challenges when working with real-world data: The Data Learning Paradigm
• Shaping Initial State Prevents Modality Competition in Multi-modal Fusion: A Two-stage Scheduling Framework via Fast Partial Information Decomposition
• Literature review on vulnerability detection using NLP technology
• Adversarial machine learning :
• A Spatially Masked Adaptive Gated Network for multimodal post-flood water extent mapping using SAR and incomplete multispectral data
• Quantifying and Alleviating the Language Prior Problem in Visual Question Answering
• AIVA: An AI-based Virtual Companion for Emotion-aware Interaction
• LightVLM: Acceleraing Large Multimodal Models with Pyramid Token Merging and KV Cache Compression
• AIM: Adaptive Intra-Network Modulation for Balanced Multimodal Learning
• New Ideas and Trends in Deep Multimodal Content Understanding: A Review
• Audio-Visual Speech Enhancement: Architectural Design and Deployment Strategies
• Multimodal deep learning approaches for single-cell multi-omics data integration
• Deep Multimodal Transfer-Learned Regression in Data-Poor Domains
• AOMD: An Analogy-aware Approach to Offensive Meme Detection on Social Media

Related

• Multimodal Deep Learning
• Multimodal Representation Learning and Fusion
• Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications
• Fusion or Confusion? Multimodal Complexity Is Not All You Need
• Multimodal Machine Learning: A Survey and Taxonomy