Developing a High-performance Framework for Speech Emotion Recognition in Naturalistic Conditions Challenge for Emotional Attribute Prediction

2025/06/12 by Thanathai Lertpetchpun, Lertpetchpun, Thanathai, Tiantian Feng +5 · 2 citations
Computer Science · Psychology · #Emotion and Mood Recognition #FOS: Computer and information sciences #Machine Learning (cs.LG) #Sentiment Analysis and Opinion Mining #Speech Recognition and Synthesis

paper · pdf · doi:10.48550/arxiv.2506.10930

openalex publication_date 2025/06/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Speech emotion recognition (SER) in naturalistic conditions presents a significant challenge for the speech processing community. Challenges include disagreement in labeling among annotators and imbalanced data distributions. This paper presents a reproducible framework that achieves superior (top 1) performance in the Emotion Recognition in Naturalistic Conditions Challenge (IS25-SER Challenge) - Task 2, evaluated on the MSP-Podcast dataset. Our system is designed to tackle the aforementioned challenges through multimodal learning, multi-task learning, and imbalanced data handling. Specifically, our best system is trained by adding text embeddings, predicting gender, and including ``Other'' (O) and ``No Agreement'' (X) samples in the training set. Our system's results secured both first and second places in the IS25-SER Challenge, and the top performance was achieved by a simple two-system ensemble.

Citations

• Finding Task-specific Subnetworks in Multi-task Spoken Language Understanding Model
• ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasets
• 1st Place Solution to Odyssey Emotion Recognition Challenge Task1: Tackling Class Imbalance Problem
• Foundation Model Assisted Automatic Speech Emotion Recognition: Transcribing, Annotating, and Augmenting
• ML-SUPERB: Multilingual Speech Universal PERformance Benchmark
• Robust Speech Recognition via Large-Scale Weak Supervision
• Fusing ASR Outputs in Joint Training for Speech Emotion Recognition
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• Emotion Recognition from Speech Using Wav2vec 2.0 Embeddings

Cited by

• MSF-SER: Enriching Acoustic Modeling with Multi-Granularity Semantics for Speech Emotion Recognition
• Smiling Regulates Emotion During Traumatic Recollection

Related

• Developing a Top-tier Framework in Naturalistic Conditions Challenge for Categorized Emotion Prediction: From Speech Foundation Models and Learning Objective to Data Augmentation and Engineering Choices
• MATER: Multi-level Acoustic and Textual Emotion Representation for Interpretable Speech Emotion Recognition
• Towards Speech Emotion Recognition "in the wild" using Aggregated Corpora and Deep Multi-Task Learning
• Speech and Text-Based Emotion Recognizer
• Sentiment-Aware Automatic Speech Recognition pre-training for enhanced Speech Emotion Recognition