What Makes Training Multi-Modal Classification Networks Hard?

2019/05/29 by Wei‐Yao Wang, Wang, Weiyao, Du Tran +3 · 31 citations
Computer Science · #Human Pose and Action Recognition #Multimodal Machine Learning Applications #Anomaly Detection Techniques and Applications

paper · pdf · doi:10.48550/arxiv.1905.12681

Abstract

Consider end-to-end training of a multi-modal vs. a single-modal network on a task with multiple input modalities: the multi-modal network receives more information, so it should match or outperform its single-modal counterpart. In our experiments, however, we observe the opposite: the best single-modal network always outperforms the multi-modal network. This observation is consistent across different combinations of modalities and on different tasks and benchmarks. This paper identifies two main causes for this performance drop: first, multi-modal networks are often prone to overfitting due to increased capacity. Second, different modalities overfit and generalize at different rates, so training them jointly with a single optimization strategy is sub-optimal. We address these two problems with a technique we call Gradient Blending, which computes an optimal blend of modalities based on their overfitting behavior. We demonstrate that Gradient Blending outperforms widely-used baselines for avoiding overfitting and achieves state-of-the-art accuracy on various tasks including human action recognition, ego-centric action recognition, and acoustic event detection.

Cited by

• Fusion or Confusion? Multimodal Complexity Is Not All You Need
• ActAvatar: Temporally-Aware Precise Action Control for Talking Avatars
• Cross-modal Prompting for Balanced Incomplete Multi-modal Emotion Recognition
• DIPOLE: Fusing Vision and Geometry for Robust Visuomotor Generalization
• Modality-Balanced Collaborative Distillation for Multi-Modal Domain Generalization
• DiffPixelFormer: Differential Pixel-Aware Transformer for RGB-D Indoor Scene Segmentation
• Context-Aware Multimodal Representation Learning for Spatio-Temporally Explicit Environmental Modelling
• Boomda: Balanced Multi-objective Optimization for Multimodal Domain Adaptation
• Balancing Multi-modal Sensor Learning via Multi-objective Optimization
• NAPS: Attention-Based Fusion of Heterogeneous Physiological Signals
• Balanced Multimodal Learning via Mutual Information
• Contribution-Guided Asymmetric Learning for Robust Multimodal Fusion under Imbalance and Noise
• Modality-Aware SAM: Sharpness-Aware-Minimization Driven Gradient Modulation for Harmonized Multimodal Learning
• Modal Aphasia: Can Unified Multimodal Models Describe Images From Memory?
• Quantifying Multimodal Imbalance: A GMM-Guided Adaptive Loss for Audio-Visual Learning
• MILES: Modality-Informed Learning Rate Scheduler for Balancing Multimodal Learning
• Revisit Modality Imbalance at the Decision Layer
• Mixup Helps Understanding Multimodal Video Better
• Data-efficient Alignment of Multimodal Sequences by Aligning Gradient Updates and Internal Feature Distributions
• Expressive and Scalable Quantum Fusion for Multimodal Learning
• OTTER: Open-Tagging via Text-Image Representation for Multi-modal Understanding
• MultiFair: Multimodal Balanced Fairness-Aware Medical Classification with Dual-Level Gradient Modulation
• MIDAS: Misalignment-based Data Augmentation Strategy for Imbalanced Multimodal Learning
• Multi-modal Data Spectrum: Multi-modal Datasets are Multi-dimensional
• Balanced Diffusion-Guided Fusion for Multimodal Remote Sensing Classification
• Shaping Initial State Prevents Modality Competition in Multi-modal Fusion: A Two-stage Scheduling Framework via Fast Partial Information Decomposition
• TIER-MoE: Trust-Informed Expert Routing via Conditional Modality Risk for Multimodal Fusion in Biomedical Classification
• GCVA: A Multiview Fusion Mechanism for Heterogeneous Data Representations
• Decoding Visual Neural Representations by Multimodal with Dynamic Balancing
• Balanced Multimodal Learning: An Unidirectional Dynamic Interaction Perspective
• AIM: Adaptive Intra-Network Modulation for Balanced Multimodal Learning

Related

• Improving Multimodal Accuracy Through Modality Pre-training and Attention
• Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)
• What Makes Multi-modal Learning Better than Single (Provably)
• Improving Discriminative Multi-Modal Learning with Large-Scale Pre-Trained Models
• Single-branch Network for Multimodal Training