Rethinking Cross-Corpus Speech Emotion Recognition Benchmarking: Are Paralinguistic Pre-Trained Representations Sufficient?

2025/09/19 by Orchid Chetia Phukan, Phukan, Orchid Chetia, Mohd Mujtaba Akhtar +10
Computer Science · #Audio and Speech Processing (eess.AS) #FOS: Electrical engineering #Speech Recognition and Synthesis #Speech and dialogue systems #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2509.16182

openalex publication_date 2025/09/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent benchmarks evaluating pre-trained models (PTMs) for cross-corpus speech emotion recognition (SER) have overlooked PTM pre-trained for paralinguistic speech processing (PSP), raising concerns about their reliability, since SER is inherently a paralinguistic task. We hypothesize that PSP-focused PTM will perform better in cross-corpus SER settings. To test this, we analyze state-of-the-art PTMs representations including paralinguistic, monolingual, multilingual, and speaker recognition. Our results confirm that TRILLsson (a paralinguistic PTM) outperforms others, reinforcing the need to consider PSP-focused PTMs in cross-corpus SER benchmarks. This study enhances benchmark trustworthiness and guides PTMs evaluations for reliable cross-corpus SER.

Citations

• SER Evals: In-domain and Out-of-domain Benchmarking for Speech Emotion Recognition
• What Does it Take to Generalize SER Model Across Datasets? A Comprehensive Benchmark
• EmoBox: Multilingual Multi-corpus Speech Emotion Recognition Toolkit and Benchmark
• EMO-SUPERB: An In-depth Look at Speech Emotion Recognition
• Are Paralinguistic Representations all that is needed for Speech Emotion Recognition?
• Transforming the Embeddings: A Lightweight Technique for Speech Emotion Recognition Tasks
• Scaling Speech Technology to 1,000+ Languages
• Robust Speech Recognition via Large-Scale Weak Supervision
• XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• Exploring Wav2vec 2.0 fine-tuning for improved speech emotion recognition
• UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Training
• SUPERB: Speech processing Universal PERformance Benchmark
• Emotion Recognition from Speech Using Wav2vec 2.0 Embeddings
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

Related

• Are Paralinguistic Representations all that is needed for Speech Emotion Recognition?
• EmoBox: Multilingual Multi-corpus Speech Emotion Recognition Toolkit and Benchmark
• ParaLBench: A Large-Scale Benchmark for Computational Paralinguistics over Acoustic Foundation Models
• Cross-Lingual Speech Emotion Recognition: Humans vs. Self-Supervised Models
• Cross Lingual Cross Corpus Speech Emotion Recognition