Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)

2022/03/23 by Yu Huang, Junyang Lin, Huang, Yu +7 · 29 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #cs.LG

paper · pdf · doi:10.48550/arxiv.2203.12221

41 pages, 2 figures

arxiv created 2022/03/23 · openalex publication_date 2022/03/23 · arxiv updated 2022/03/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Despite the remarkable success of deep multi-modal learning in practice, it has not been well-explained in theory. Recently, it has been observed that the best uni-modal network outperforms the jointly trained multi-modal network, which is counter-intuitive since multiple signals generally bring more information. This work provides a theoretical explanation for the emergence of such performance gap in neural networks for the prevalent joint training framework. Based on a simplified data distribution that captures the realistic property of multi-modal data, we prove that for the multi-modal late-fusion network with (smoothed) ReLU activation trained jointly by gradient descent, different modalities will compete with each other. The encoder networks will learn only a subset of modalities. We refer to this phenomenon as modality competition. The losing modalities, which fail to be discovered, are the origins where the sub-optimality of joint training comes from. Experimentally, we illustrate that modality competition matches the intrinsic behavior of late-fusion joint training.

Cited by

• DIPOLE: Fusing Vision and Geometry for Robust Visuomotor Generalization
• Bird-MML: A Multimodal Dataset for Audio-Visual Complementarity
• Quantifying Modality Contributions via Disentangling Multimodal Representations
• Balancing Multi-modal Sensor Learning via Multi-objective Optimization
• NAPS: Attention-Based Fusion of Heterogeneous Physiological Signals
• Contribution-Guided Asymmetric Learning for Robust Multimodal Fusion under Imbalance and Noise
• Modality-Aware SAM: Sharpness-Aware-Minimization Driven Gradient Modulation for Harmonized Multimodal Learning
• A Video Is Not Worth a Thousand Words
• DeepfakeBench-MM: A Comprehensive Benchmark for Multimodal Deepfake Detection
• MILES: Modality-Informed Learning Rate Scheduler for Balancing Multimodal Learning
• OS-HGAdapter: Open Semantic Hypergraph Adapter for Large Language Models Assisted Entropy-Enhanced Image-Text Alignment
• MCE: Towards a General Framework for Handling Missing Modalities under Imbalanced Missing Rates
• Multi-modal Data Spectrum: Multi-modal Datasets are Multi-dimensional
• Robust Multi-Omics Integration from Incomplete Modalities Significantly Improves Prediction of Alzheimer's Disease
• Shaping Initial State Prevents Modality Competition in Multi-modal Fusion: A Two-stage Scheduling Framework via Fast Partial Information Decomposition
• TIER-MoE: Trust-Informed Expert Routing via Conditional Modality Risk for Multimodal Fusion in Biomedical Classification
• AIM: Adaptive Intra-Network Modulation for Balanced Multimodal Learning
• SEAM: Semantically Equivalent Across Modalities Benchmark for Vision-Language Models
• Partitioner Guided Modal Learning Framework
• Bridging Domain Generalization to Multimodal Domain Generalization via Unified Representations
• Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoders
• G²D: Boosting Multimodal Learning with Gradient-Guided Distillation
• Cause-Effect Driven Optimization for Robust Medical Visual Question Answering with Language Biases
• Improving Multimodal Learning Balance and Sufficiency through Data Remixing
• RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer
• QoQ-Med: Building Multimodal Clinical Foundation Models with Domain-Aware GRPO Training
• RMMSS: Towards Advanced Robust Multi-Modal Semantic Segmentation with Hybrid Prototype Distillation and Feature Selection
• LLM Features Can Hurt GNNs: Concatenation Interference on Homophilous Graph Benchmarks
• C³PO: Evaluating Cross-Modal Composition and Counterfactual Performance in Omnimodal Models

Related

• What Makes Multi-modal Learning Better than Single (Provably)
• Boosting Multi-modal Model Performance with Adaptive Gradient Modulation
• What Makes Training Multi-Modal Classification Networks Hard?
• Shaping Initial State Prevents Modality Competition in Multi-modal Fusion: A Two-stage Scheduling Framework via Fast Partial Information Decomposition
• Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks