Speaker Style-Aware Phoneme Anchoring for Improved Cross-Lingual Speech Emotion Recognition

2025/09/19 by Upadhyay, Shreya G., Busso, Carlos, Lee, Chi-Chun
#Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2509.20373

Abstract

Cross-lingual speech emotion recognition (SER) remains a challenging task due to differences in phonetic variability and speaker-specific expressive styles across languages. Effectively capturing emotion under such diverse conditions requires a framework that can align the externalization of emotions across different speakers and languages. To address this problem, we propose a speaker-style aware phoneme anchoring framework that aligns emotional expression at the phonetic and speaker levels. Our method builds emotion-specific speaker communities via graph-based clustering to capture shared speaker traits. Using these groups, we apply dual-space anchoring in speaker and phonetic spaces to enable better emotion transfer across languages. Evaluations on the MSP-Podcast (English) and BIIC-Podcast (Taiwanese Mandarin) corpora demonstrate improved generalization over competitive baselines and provide valuable insights into the commonalities in cross-lingual emotion representation.

Citations

• Mouth Articulation-Based Anchoring for Improved Cross-Corpus Speech Emotion Recognition
• A Layer-Anchoring Strategy for Enhancing Cross-Lingual Speech Emotion Recognition
• EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animation
• FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversion
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversion
• HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
• Compact Graph Architecture for Speech Emotion Recognition

Related

• A Layer-Anchoring Strategy for Enhancing Cross-Lingual Speech Emotion Recognition
• Large Language Models Meet Contrastive Learning: Zero-Shot Emotion Recognition Across Languages
• METTS: Multilingual Emotional Text-to-Speech by Cross-speaker and Cross-lingual Emotion Transfer
• Leveraging Cross-Attention Transformer and Multi-Feature Fusion for Cross-Linguistic Speech Emotion Recognition
• EM2LDL: A Multilingual Speech Corpus for Mixed Emotion Recognition through Label Distribution Learning