Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects
2015/08/20 by Dana Lahat, Tulay Adali, Tülay Adalı +1 · 1,265 citations
Computer Science · Engineering · #Blind Source Separation Techniques #Computer science #Data science #Remote-Sensing Image Classification #Text and Document Classification Technologies
paper · open access · doi:10.1109/jproc.2015.2460697
published in Proceedings of the IEEE 103(9), 1449-1477 (Institute of Electrical and Electronics Engineers)
openalex publication_date 2015/08/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/30
Abstract
In various disciplines, information about the same phenomenon can be acquired from different types of detectors, at different conditions, in multiple experiments or subjects, among others. We use the term “modality” for each such acquisition framework. Due to the rich characteristics of natural phenomena, it is rare that a single modality provides complete knowledge of the phenomenon of interest. The increasing availability of several modalities reporting on the same system introduces new degrees of freedom, which raise questions beyond those related to exploiting each modality separately. As we argue, many of these questions, or “challenges,” are common to multiple domains. This paper deals with two key issues: “why we need data fusion” and “how we perform it.” The first issue is motivated by numerous examples in science and technology, followed by a mathematical framework that showcases some of the benefits that data fusion provides. In order to address the second issue, “diversity” is introduced as a key concept, and a number of data-driven solutions based on matrix and tensor decompositions are discussed, emphasizing how they account for diversity across the data sets. The aim of this paper is to provide the reader, regardless of his or her community of origin, with a taste of the vastness of the field, the prospects, and the opportunities that it holds.
Citations
Cited by
- Data Collection in Multimodal Language and Communication Research: A Flexible Decision Framework
- Comprehensive Analysis on Machine Learning Approaches for Interpretable and Stable Soft Sensors
- An explainable multi-modal feature fusion method based on correlation-compensated Choquet integration
- Variational Low-rank Tensor Decomposition for Multisubject Spatiotemporal Data Analysis
- Multimodal Latent Variable Analysis
- Deep-HOSeq: Deep Higher Order Sequence Fusion for Multimodal Sentiment Analysis
- Rank-based linkage I: triplet comparisons and oriented simplicial complexes
- How do kernel-based sensor fusion algorithms behave under high dimensional noise?
- Protecting Big Data Privacy Using Randomized Tensor Network Decomposition and Dispersed Tensor Computation
- Emerging Biometrics: Deep Inference and Other Computational Intelligence
- Quantum mechanics-based multitensor AI/ML uniquely able to discover, validate, and interpret predictors from small-cohort noisy high-dimensional multiomic data
- Deep Multimodal Learning: A Survey on Recent Advances and Trends
- Tensor Decomposition for Signal Processing and Machine Learning
- MMDF2018 Workshop Report
- GAL: Gradient Assisted Learning for Decentralized Multi-Organization Collaborations
- A comprehensive review of techniques, algorithms, advancements, challenges, and clinical applications of multi-modal medical image fusion for improved diagnosis
- Fuzzy-ViT: A Deep Neuro-Fuzzy System for Cross-Domain Transfer Learning From Large-Scale General Data to Medical Image
- From Adaptation to Intelligence: A Systematic Review of Data, Strategies, and Impact in Personalized VR
- FlyAwareV2: A Multimodal Cross-Domain UAV Dataset for Urban Scene Understanding
- Progressive multi-fidelity learning with neural networks for physical system predictions
- Collaborative Multi-Robot Systems for Search and Rescue: Coordination and Perception
- A Multisensory Learning Architecture for Rotation-invariant Object Recognition
- Artificial Intelligence in Multimodal Learning Process Analytics
- PHASE: Physics-Integrated, Heterogeneity-Aware Surrogates for Scientific Simulations
- Non-Invasive Brain-Computer Interfaces: Converging Frontiers in Neural Signal Decoding and Flexible Bioelectronics Integration
- Multimodal Medical Image Classification via Synergistic Learning Pre-training
- HetMAML: Task-Heterogeneous Model-Agnostic Meta-Learning for Few-Shot Learning Across Modalities
- Efficient Semiparametric Inference for Distributed Data with Blockwise Missingness
- Multimodal learning of melt pool dynamics in laser powder bed fusion
- When Vision-Language Model (VLM) Meets Beam Prediction: A Multimodal Contrastive Learning Framework
- WaveletInception Networks for Drive-by Vibration-Based Infrastructure Health Monitoring
- ATCTrack: Aligning Target-Context Cues with Dynamic Target States for Robust Vision-Language Tracking
- Multi-Modal Neuroimaging Analysis and Visualization Tool (MMVT)
- Machine Learning Techniques in Adaptive and Personalized Systems for Health and Wellness
- Computer-Assisted Analysis of Biomedical Images
- Multimodal Emotion Recognition Using Deep Canonical Correlation Analysis
- Introducing machine‐learning‐based data fusion methods for analyzing multimodal data: An application of measuring trustworthiness of microenterprises
- FuDoBa: Fusing Document and Knowledge Graph-based Representations with Bayesian Optimisation
- Multi-task Learning with High-Dimensional Noisy Images
- A Survey of Pun Generation: Datasets, Evaluations and Methodologies
- Common and Distinct Components in Data Fusion
- A Survey on Deep Learning for Multimodal Data Fusion
- Towards a Unified Textual Graph Framework for Spectral Reasoning via Physical and Chemical Information Fusion
- ASCII: ASsisted Classification with Ignorance Interchange
- Diffusion-based nonlinear filtering for multimodal data fusion with application to sleep stage assessment
- Assisted Learning: A Framework for Multi-Organization Learning
- Generalizing from a Few Examples: A Survey on Few-Shot Learning
- The Tensor Track VII: From Quantum Gravity to Artificial Intelligence
- Explainable AI for Intelligence Augmentation in Multi-Domain Operations
- Latent common manifold learning with alternating diffusion: analysis and applications
- Fusing Cross-Domain Knowledge from Multimodal Data to Solve Problems in the Physical World
- Rough Set Microbiome Characterisation
- A Comprehensive Review of Techniques, Algorithms, Advancements, Challenges, and Clinical Applications of Multi-modal Medical Image Fusion for Improved Diagnosis
- Multimodal domain adaptation under label shift and blockwise missing modalities
- A novel multimodal approach for hybrid brain-computer interface
- Multimodal Data Fusion based on the Global Workspace Theory
- Cloud Is Closer Than It Appears: Revisiting the Tradeoffs of Distributed Real-Time Inference
- Coarse-to-Fine Registration of Airborne LiDAR Data and Optical Imagery on Urban Scenes
- Multi-Level Sensor Fusion with Deep Learning
- Hadamard product in deep learning: Introduction, Advances and Challenges
- Sensor Fusion and Smart Sensor in Sports and Biomedical Applications. [europepmc]
- Looking beyond the cancer cell for effective drug combinations. [europepmc]
- Schizophrenia Shows Disrupted Links between Brain Volume and Dynamic Functional Connectivity. [europepmc]
- Application of Data Mining to "Big Data" Acquired in Audiology: Principles and Potential. [europepmc]
- Robust Fusion of LiDAR and Wide-Angle Camera Data for Autonomous Mobile Robots. [europepmc]
- Detecting Mistakes in CPR Training with Multimodal Data and Neural Networks. [europepmc]
- In search of multimodal brain alterations in Alzheimer's and Binswanger's disease. [europepmc]
- High Definition 3D Map Creation Using GNSS/IMU/LiDAR Sensor Integration to Support Autonomous Vehicle Navigation. [europepmc]
- A critical review of recent trends, and a future perspective of optical spectroscopy as PAT in biopharmaceutical downstream processing. [europepmc]
- What Can Computational Models Learn From Human Selective Attention? A Review From an Audiovisual Unimodal and Crossmodal Perspective. [europepmc]
- Survey of Datafusion Techniques for Laser and Vision Based Sensor Integration for Autonomous Navigation. [europepmc]
- Multi-dimensional predictions of psychotic symptoms via machine learning. [europepmc]
- Hand-Gesture Recognition Based on EMG and Event-Based Camera Sensor Fusion: A Benchmark in Neuromorphic Computing. [europepmc]
- Status and Recommendations of Technological and Data-Driven Innovations in Cancer Care: Focus Group Study. [europepmc]
- The Human Organism as an Integrated Interaction Network: Recent Conceptual and Methodological Challenges. [europepmc]
- Computer-aided diagnosis of hepatocellular carcinoma fusing imaging and structured health data. [europepmc]
- A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets. [europepmc]
- Employing Multimodal Machine Learning for Stress Detection. [europepmc]
- Structured sparse multiset canonical correlation analysis of simultaneous fNIRS and EEG provides new insights into the human action-observation network. [europepmc]
- Multimodal machine learning in precision health: A scoping review. [europepmc]
- Multimodal Data Integration to Predict Severe Acute Oral Mucositis of Nasopharyngeal Carcinoma Patients Following Radiation Therapy. [europepmc]
- A systematic review of neurophysiological sensing for the assessment of acute pain. [europepmc]
- Emerging Trends in Machine Learning: A Polymer Perspective. [europepmc]
- Machine learning with multimodal neuroimaging data to classify stages of Alzheimer's disease: a systematic review and meta-analysis. [europepmc]
- A machine learning approach for multimodal data fusion for survival prediction in cancer patients. [europepmc]
Related