Provable Dynamic Fusion for Low-Quality Multimodal Data

2023/06/03 by Qingyang Zhang, Haitao Wu, Zhang, Qingyang +11 · 12 citations
Engineering · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Remote-Sensing Image Classification

paper · pdf · doi:10.48550/arxiv.2306.02050

openalex publication_date 2023/06/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The inherent challenge of multimodal fusion is to precisely capture the cross-modal correlation and flexibly conduct cross-modal interaction. To fully release the value of each modality and mitigate the influence of low-quality multimodal data, dynamic multimodal fusion emerges as a promising learning paradigm. Despite its widespread use, theoretical justifications in this field are still notably lacking. Can we design a provably robust multimodal fusion method? This paper provides theoretical understandings to answer this question under a most popular multimodal fusion framework from the generalization perspective. We proceed to reveal that several uncertainty estimation solutions are naturally available to achieve robust multimodal fusion. Then a novel multimodal fusion framework termed Quality-aware Multimodal Fusion (QMF) is proposed, which can improve the performance in terms of classification accuracy and model robustness. Extensive experimental results on multiple benchmarks can support our findings.

Cited by

• A-QCF-Net: An Adaptive Quaternion Cross-Fusion Network for Multimodal Liver Tumor Segmentation from Unpaired Datasets
• FINE: Factorized multimodal sentiment analysis via mutual INformation Estimation
• Robust Multimodal Sentiment Analysis of Image-Text Pairs by Distribution-Based Feature Recovery and Fusion
• MULTIBENCH++: A Unified and Comprehensive Multimodal Fusion Benchmarking Across Specialized Domains
• Contribution-Guided Asymmetric Learning for Robust Multimodal Fusion under Imbalance and Noise
• Multimodal Negative Learning
• Fusion Meets Diverse Conditions: A High-diversity Benchmark and Baseline for UAV-based Multimodal Object Detection with Condition Cues
• TIER-MoE: Trust-Informed Expert Routing via Conditional Modality Risk for Multimodal Fusion in Biomedical Classification
• Pseudo-D: Informing Multi-View Uncertainty Estimation with Calibrated Neural Training Dynamics
• Multi-modal Uncertainty Robust Tree Cover Segmentation For High-Resolution Remote Sensing Images
• AIM: Adaptive Intra-Network Modulation for Balanced Multimodal Learning
• Mitigating Modality Quantity and Quality Imbalance in Multimodal Online Federated Learning

Related

• Multimodal Fusion on Low-quality Data: A Comprehensive Survey
• Predictive Dynamic Fusion
• Deep Equilibrium Multimodal Fusion
• Dynamic Multimodal Fusion
• Fusion or Confusion? Multimodal Complexity Is Not All You Need