Multimodal Emotion Recognition in Conversations: A Survey of Methods, Trends, Challenges and Prospects

2025/05/26 by Changxu Wu, Wu, Chengyan, Yunna Cai +13 · 2 citations
Psychology · Computer Science · #Emotion and Mood Recognition #Sentiment Analysis and Opinion Mining #Speech and dialogue systems

paper · pdf · doi:10.48550/arxiv.2505.20511

Abstract

While text-based emotion recognition methods have achieved notable success, real-world dialogue systems often demand a more nuanced emotional understanding than any single modality can offer. Multimodal Emotion Recognition in Conversations (MERC) has thus emerged as a crucial direction for enhancing the naturalness and emotional understanding of human-computer interaction. Its goal is to accurately recognize emotions by integrating information from various modalities such as text, speech, and visual signals. This survey offers a systematic overview of MERC, including its motivations, core tasks, representative methods, and evaluation strategies. We further examine recent trends, highlight key challenges, and outline future directions. As interest in emotionally intelligent systems grows, this survey provides timely guidance for advancing MERC research.

Citations

• Akan Cinematic Emotions (ACE): A Multimodal Multi-party Dataset for Emotion Recognition in Movie Dialogues
• Cross-modal Context Fusion and Adaptive Graph Convolutional Network for Multimodal Conversational Emotion Recognition
• Effective Context Modeling Framework for Emotion Recognition in Conversations
• Multimodal Alignment and Fusion: A Survey
• Smile upon the Face but Sadness in the Eyes: Emotion Recognition based on Facial Expressions and Eye Behaviors
• Beyond Silent Letters: Amplifying LLMs in Emotion Recognition with Vocal Nuances
• Learning Modality Knowledge Alignment for Cross-Modality Transfer
• Multimodal Fusion on Low-quality Data: A Comprehensive Survey
• Revisiting Multimodal Emotion Recognition in Conversation from the Perspective of Graph Spectrum
• LaERC-S: Improving LLM-based Emotion Recognition in Conversation with Speaker Characteristics
• Text-centric Alignment for Multi-Modality Learning
• Graph Neural Networks in EEG-based Emotion Recognition: A Survey
• FourierGNN: Rethinking Multivariate Time Series Forecasting from a Pure Graph Perspective
• DialogueLLM: Context and Emotion Knowledge-Tuned Large Language Models for Emotion Recognition in Conversations
• UniSA: Unified Generative Framework for Sentiment Analysis
• Revisiting Graph Contrastive Learning from the Perspective of Graph Spectrum
• M2FNet: Multi-modal Fusion Network for Emotion Recognition in Conversation
• EmoCaps: Emotion Capsule based Model for Conversational Emotion Recognition
• GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversation
• Finetuned Language Models Are Zero-Shot Learners
• MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation
• Multi-Modal Emotion Detection with Transfer Learning
• COSMIC: COmmonSense knowledge for eMotion Identification in Conversations
• Real-Time Emotion Recognition via Attention Gated Hierarchical Memory Network
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• Hierarchical Gating Networks for Sequential Recommendation
• Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation
• Emotion Recognition in Conversation: Research Challenges, Datasets, and\n Recent Advances
• DialogueRNN: An Attentive RNN for Emotion Detection in Conversations
• MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations
• Hypergraph Neural Networks
• Representation Learning with Contrastive Predictive Coding
• EmotionLines: An Emotion Corpus of Multi-Party Conversations
• Survey on Emotional Body Gesture Recognition
• Emotion Detection on TV Show Transcripts with Sequence-based Convolutional Neural Networks
• Attention Is All You Need
• Densely Connected Convolutional Networks
• Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks
• Convolutional Neural Networks for Sentence Classification
• Multimodal fusion for multimedia analysis: a survey
• Long Short-Term Memory
• The Graph Neural Network Model

Cited by

• Decoding Student Minds: Leveraging Conversational Agents for Psychological and Learning Analysis
• Learning More with Less: Self-Supervised Approaches for Low-Resource Speech Emotion Recognition
• Narrative-Centered Emotional Reflection: An Early Prototype for AI-Supported Emotional Self-Reflection

Related

• GatedxLSTM: A Multimodal Affective Computing Approach for Emotion Recognition in Conversations
• Shapes of Emotions: Multimodal Emotion Recognition in Conversations via Emotion Shifts
• Emotion Recognition in Conversation: Research Challenges, Datasets, and\n Recent Advances
• A Comprehensive Survey on Multi-modal Conversational Emotion Recognition with Deep Learning
• MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations