Improving Multimodal Accuracy Through Modality Pre-training and Attention

2020/11/11 by Aya Abdelsalam Ismail, Ismail, Aya Abdelsalam, Md. Mahmudul Hasan +4 · 2 citations
Computer Science · Psychology · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Language, Metaphor, and Cognition #Second Language Acquisition and Learning #Speech and dialogue systems #cs.AI

paper · pdf · doi:10.48550/arxiv.2011.06102

arxiv created 2020/11/11 · openalex publication_date 2020/11/11 · arxiv updated 2020/11/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Training a multimodal network is challenging and it requires complex architectures to achieve reasonable performance. We show that one reason for this phenomena is the difference between the convergence rate of various modalities. We address this by pre-training modality-specific sub-networks in multimodal architectures independently before end-to-end training of the entire network. Furthermore, we show that the addition of an attention mechanism between sub-networks after pre-training helps identify the most important modality during ambiguous scenarios boosting the performance. We demonstrate that by performing these two tricks a simple network can achieve similar performance to a complicated architecture that is significantly more expensive to train on multiple tasks including sentiment analysis, emotion recognition, and speaker trait recognition.

Citations

• A Survey on Multi-view Learning
• Reasoning about Entailment with Neural Attention
• Sequence to Sequence Learning with Neural Networks
• Bidirectional recurrent neural networks
• Understanding Neural Networks through Representation Erasure
• Efficient Low-rank Multimodal Fusion with Modality-Specific Factors
• Learning Factorized Multimodal Representations
• Memory Fusion Network for Multi-view Sequential Learning

Cited by

• MedM2T: A MultiModal Framework for Time-Aware Modeling with Electronic Health Record and Electrocardiogram Data
• Modality-Aware SAM: Sharpness-Aware-Minimization Driven Gradient Modulation for Harmonized Multimodal Learning

Related

• What Makes Training Multi-Modal Classification Networks Hard?
• Explore the Limits of Omni-modal Pretraining at Scale
• Modality Invariant Multimodal Learning to Handle Missing Modalities: A Single-Branch Approach
• Single-branch Network for Multimodal Training
• MILES: Modality-Informed Learning Rate Scheduler for Balancing Multimodal Learning