Enhancing multimodal cooperation via sample-level modality valuation

2023/09/12 by Yake Wei, Ruoxuan Feng, Wei, Yake +5 · 13 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimedia (cs.MM)

paper · pdf · doi:10.48550/arxiv.2309.06255

openalex publication_date 2023/09/12 · openalex created_date 2023/09/14 · openalex updated_date 2026/07/28

Abstract

One primary topic of multimodal learning is to jointly incorporate heterogeneous information from different modalities. However most models often suffer from unsatisfactory multimodal cooperation which cannot jointly utilize all modalities well. Some methods are proposed to identify and enhance the worse learnt modality but they are often hard to provide the fine-grained observation of multimodal cooperation at sample-level with theoretical support. Hence it is essential to reasonably observe and improve the fine-grained cooperation between modalities especially when facing realistic scenarios where the modality discrepancy could vary across different samples. To this end we introduce a sample-level modality valuation metric to evaluate the contribution of each modality for each sample. Via modality valuation we observe that modality discrepancy indeed could be different at sample-level beyond the global contribution discrepancy at dataset-level. We further analyze this issue and improve cooperation between modalities at sample-level by enhancing the discriminative ability of low-contributing modalities in a targeted manner. Overall our methods reasonably observe the fine-grained uni-modal contribution and achieve considerable improvement. The source code and dataset are available at https://github.com/GeWu-Lab/Valuate-and-Enhance-Multimodal-Cooperation.

Cited by

• Cross-modal Prompting for Balanced Incomplete Multi-modal Emotion Recognition
• Modality-Balanced Collaborative Distillation for Multi-Modal Domain Generalization
• Distilling Cross-Modal Knowledge via Feature Disentanglement
• Quantifying Modality Contributions via Disentangling Multimodal Representations
• Unveiling Modality Bias: Automated Sample-Specific Analysis for Multimodal Misinformation Benchmarks
• Multimodal Negative Learning
• MCE: Towards a General Framework for Handling Missing Modalities under Imbalanced Missing Rates
• SAMSOD: Rethinking SAM Optimization for RGB-T Salient Object Detection
• MIDAS: Misalignment-based Data Augmentation Strategy for Imbalanced Multimodal Learning
• Shaping Initial State Prevents Modality Competition in Multi-modal Fusion: A Two-stage Scheduling Framework via Fast Partial Information Decomposition
• Regularizing modality contribution drift in multimodal continual learning
• Decoding Visual Neural Representations by Multimodal with Dynamic Balancing
• Balanced Multimodal Learning: An Unidirectional Dynamic Interaction Perspective

Related

• Quantifying Modality Contributions via Disentangling Multimodal Representations
• Greedy Modality Selection via Approximate Submodular Maximization
• Rebalanced Multimodal Learning with Data-aware Unimodal Sampling
• Balanced Multimodal Learning via Mutual Information
• Modality Invariant Multimodal Learning to Handle Missing Modalities: A Single-Branch Approach