Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis

2021/02/09 by Wenmeng Yu, Yu, Wenmeng, Hua Xu +5 · 27 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Sentiment Analysis and Opinion Mining #Text and Document Classification Technologies

paper · pdf · doi:10.48550/arxiv.2102.04830

openalex publication_date 2021/02/09 · openalex created_date 2021/07/05 · openalex updated_date 2026/07/28

Abstract

Representation Learning is a significant and challenging task in multimodal learning. Effective modality representations should contain two parts of characteristics: the consistency and the difference. Due to the unified multimodal annotation, existing methods are restricted in capturing differentiated information. However, additional uni-modal annotations are high time- and labor-cost. In this paper, we design a label generation module based on the self-supervised learning strategy to acquire independent unimodal supervisions. Then, joint training the multi-modal and uni-modal tasks to learn the consistency and difference, respectively. Moreover, during the training stage, we design a weight-adjustment strategy to balance the learning progress among different subtasks. That is to guide the subtasks to focus on samples with a larger difference between modality supervisions. Last, we conduct extensive experiments on three public multimodal baseline datasets. The experimental results validate the reliability and stability of auto-generated unimodal supervisions. On MOSI and MOSEI datasets, our method surpasses the current state-of-the-art methods. On the SIMS dataset, our method achieves comparable performance than human-annotated unimodal labels. The full codes are available at https://github.com/thuiar/Self-MM.

Cited by

• Smile on the Face, Sadness in the Eyes: Bridging the Emotion Gap with a Multimodal Dataset of Eye and Facial Behaviors
• MIDG: Mixture of Invariant Experts with knowledge injection for Domain Generalization in Multimodal Sentiment Analysis
• DashFusion: Dual-stream Alignment with Hierarchical Bottleneck Fusion for Multimodal Sentiment Analysis
• Augmenting Intra-Modal Understanding in MLLMs for Robust Multimodal Keyphrase Generation
• FINE: Factorized multimodal sentiment analysis via mutual INformation Estimation
• Senti-iFusion: An Integrity-centered Hierarchical Fusion Framework for Multimodal Sentiment Analysis under Uncertain Modality Missingness
• Robust Multimodal Sentiment Analysis via Double Information Bottleneck
• Improving Multimodal Sentiment Analysis via Modality Optimization and Dynamic Primary Modality Selection
• QiNN-QJ: A Quantum-inspired Neural Network with Quantum Jump for Multimodal Sentiment Analysis
• MCIHN: A Hybrid Network Model Based on Multi-path Cross-modal Interaction for Multimodal Emotion Recognition
• FSRF: Factorization-guided Semantic Recovery for Incomplete Multimodal Sentiment Analysis
• Calibrating Multimodal Consensus for Emotion Recognition
• Edu-EmotionNet: Cross-Modality Attention Alignment with Temporal Feedback Loops
• Beyond Simple Fusion: Adaptive Gated Fusion for Robust Multimodal Sentiment Analysis
• MIDAS: Misalignment-based Data Augmentation Strategy for Imbalanced Multimodal Learning
• Towards Minimal Causal Representations for Human Multimodal Language Understanding
• Semantic-Aligned Structural Abstraction for Multimodal Sentiment Analysis
• Structures Meet Semantics: Multimodal Fusion via Graph Contrastive Learning
• On Transferring, Merging, and Splitting Task-Oriented Network Digital Twins
• Uncertainty-Aware Collaborative System of Large and Small Models for Multimodal Sentiment Analysis
• Towards Multimodal Sentiment Analysis via Contrastive Cross-modal Retrieval Augmentation and Hierachical Prompts
• Disentangling Bias by Modeling Intra- and Inter-modal Causal Attention for Multimodal Sentiment Analysis
• Weakly Supervised Multimodal Temporal Forgery Localization via Multitask Learning
• Hierarchical MoE: Continuous Multimodal Emotion Recognition with Incomplete and Asynchronous Inputs
• Graph-based Interaction Augmentation Network for Robust Multimodal Sentiment Analysis
• MoLAN: A Unified Modality-Aware Noise Dynamic Editing Framework for Multimodal Sentiment Analysis
• Hybrid CNN-Mamba Enhancement Network for Robust Multimodal Sentiment Analysis

Related

• Improving the Modality Representation with Multi-View Contrastive Learning for Multimodal Sentiment Analysis
• Meta-Learn Unimodal Signals with Weak Supervision for Multimodal Sentiment Analysis
• Multi-Modality Collaborative Learning for Sentiment Analysis
• Toward Robust Multimodal Learning using Multimodal Foundational Models
• A survey on Self Supervised learning approaches for improving Multimodal representation learning