Multimodal Emotion Recognition and Sentiment Analysis in Multi-Party Conversation Contexts

2025/03/09 by Farhadipour, Aref, Ranjbar, Hossein, Chapariniya, Masoumeh +3 · 1 citation
#Audio and Speech Processing (eess.AS) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2503.06805

Abstract

Emotion recognition and sentiment analysis are pivotal tasks in speech and language processing, particularly in real-world scenarios involving multi-party, conversational data. This paper presents a multimodal approach to tackle these challenges on a well-known dataset. We propose a system that integrates four key modalities/channels using pre-trained models: RoBERTa for text, Wav2Vec2 for speech, a proposed FacialNet for facial expressions, and a CNN+Transformer architecture trained from scratch for video analysis. Feature embeddings from each modality are concatenated to form a multimodal vector, which is then used to predict emotion and sentiment labels. The multimodal system demonstrates superior performance compared to unimodal approaches, achieving an accuracy of 66.36% for emotion recognition and 72.15% for sentiment analysis.

Cited by

• Towards Language-Independent Face-Voice Association with Multimodal Foundation Models

Related

• Benchmarking Multimodal Sentiment Analysis
• Rethinking Multimodal Sentiment Analysis: A High-Accuracy, Simplified Fusion Architecture
• Multimodal Utterance-level Affect Analysis using Visual, Audio and Text Features
• GCM-Net: Graph-enhanced Cross-Modal Infusion with a Metaheuristic-Driven Network for Video Sentiment and Emotion Analysis
• Cross-modal Context Fusion and Adaptive Graph Convolutional Network for Multimodal Conversational Emotion Recognition