Representation Decomposition for Learning Similarity and Contrastness Across Modalities for Affective Computing

2025/06/08 by Yuanhe Tian, Tian, Yuanhe, Pengsen Cheng +7 · 2 citations
Computer Science · Psychology · #Computation and Language (cs.CL) #Emotion and Mood Recognition #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Sentiment Analysis and Opinion Mining

paper · pdf · doi:10.48550/arxiv.2506.07086

openalex publication_date 2025/06/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Multi-modal affective computing aims to automatically recognize and interpret human attitudes from diverse data sources such as images and text, thereby enhancing human-computer interaction and emotion understanding. Existing approaches typically rely on unimodal analysis or straightforward fusion of cross-modal information that fail to capture complex and conflicting evidence presented across different modalities. In this paper, we propose a novel LLM-based approach for affective computing that explicitly deconstructs visual and textual representations into shared (modality-invariant) and modality-specific components. Specifically, our approach firstly encodes and aligns input modalities using pre-trained multi-modal encoders, then employs a representation decomposition framework to separate common emotional content from unique cues, and finally integrates these decomposed signals via an attention mechanism to form a dynamic soft prompt for a multi-modal LLM. Extensive experiments on three representative tasks for affective computing, namely, multi-modal aspect-based sentiment analysis, multi-modal emotion analysis, and hateful meme detection, demonstrate the effectiveness of our approach, which consistently outperforms strong baselines and state-of-the-art models.

Citations

• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• Large Language Models Meet Text-Centric Multimodal Sentiment Analysis: A Survey
• Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
• Decoding the Underlying Meaning of Multimodal Hateful Memes
• Prompting for Multimodal Hateful Meme Classification
• Multimodal Feature Extraction for Memes Sentiment Classification
• Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
• Memes in the Wild: Assessing the Generalizability of the Hateful Memes\n Challenge Dataset
• Multimodal Few-Shot Learning with Frozen Language Models
• AOMD: An Analogy-aware Approach to Offensive Meme Detection on Social Media
• AOMD: An analogy-aware approach to offensive meme detection on social media
• ZEN 2.0: Continue Training and Adaption for N-gram Enhanced Text Encoders
• Targeted aspect based multimodal sentiment analysis:an attention capsule extraction and multi-head fusion network
• Learning Transferable Visual Models From Natural Language Supervision
• Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challenge
• Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful Memes
• Multimodal Learning for Hateful Memes Detection
• UPB at SemEval-2020 Task 8: Joint Textual and Visual Modeling in a\n Multi-Task Learning Architecture for Memotion Analysis
• IITK at SemEval-2020 Task 8: Unimodal and Bimodal Sentiment Analysis of\n Internet Memes
• The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• Incorporating Context and External Knowledge for Pronoun Coreference Resolution
• hyperdoc2vec: Distributed Representations of Hypertext Documents
• Deep contextualized word representations
• Attention Is All You Need
• Multimodal Machine Learning: A Survey and Taxonomy
• Multimodal Machine Learning: A Survey and Taxonomy
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal\n Networks
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Efficient Estimation of Word Representations in Vector Space
• Robust Principal Component Analysis?
• Robust principal component analysis?
• DualKanbaFormer: An Efficient Selective Sparse Framework for Multimodal Aspect-based Sentiment Analysis
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• Recurrent Visual Feature Extraction and Stereo Attentions for CT Report Generation
• Large Language Models Enhanced by Plug and Play Syntactic Knowledge for Aspect-based Sentiment Analysis

Related

• GIA-MIC: Multimodal Emotion Recognition with Gated Interactive Attention and Modality-Invariant Learning Constraints
• EffMulti: Efficiently Modeling Complex Multimodal Interactions for Emotion Analysis
• Adversarial Representation with Intra-Modal and Inter-Modal Graph Contrastive Learning for Multimodal Emotion Recognition
• Recent Trends of Multimodal Affective Computing: A Survey from NLP Perspective
• Joyful: Joint Modality Fusion and Graph Contrastive Learning for Multimodal Emotion Recognition