Few-shot Personalization via In-Context Learning for Speech Emotion Recognition based on Speech-Language Model

2025/09/10 by Mana Ihori, Taiga Yamane, Ihori, Mana +13
Computer Science · Psychology · #Audio and Speech Processing (eess.AS) #Emotion and Mood Recognition #FOS: Electrical engineering #Sentiment Analysis and Opinion Mining #Speech Recognition and Synthesis #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2509.08344

openalex publication_date 2025/09/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper proposes a personalization method for speech emotion recognition (SER) through in-context learning (ICL). Since the expression of emotions varies from person to person, speaker-specific adaptation is crucial for improving the SER performance. Conventional SER methods have been personalized using emotional utterances of a target speaker, but it is often difficult to prepare utterances corresponding to all emotion labels in advance. Our idea to overcome this difficulty is to obtain speaker characteristics by conditioning a few emotional utterances of the target speaker in ICL-based inference. ICL is a method to perform unseen tasks by conditioning a few input-output examples through inference in large language models (LLMs). We meta-train a speech-language model extended from the LLM to learn how to perform personalized SER via ICL. Experimental results using our newly collected SER dataset demonstrate that the proposed method outperforms conventional methods.

Citations

• In-Context Learning Boosts Speech Recognition via Human-like Adaptation to Speakers and Language Varieties
• Steering Language Model to Stable Speech Emotion Recognition via Contextual Perception and Chain of Thought
• Qwen2-Audio Technical Report
• Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities
• SECap: Speech Emotion Captioning with Large Language Model
• COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learning
• SALMONN: Towards Generic Hearing Abilities for Large Language Models
• Exploring In-Context Learning of Textless Speech Language Model for Speech Classification Tasks
• SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translation
• Can Whisper perform speech-based in-context learning?
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• Exploring speaker enrolment for few-shot personalisation in emotional vocalisation prediction
• UL2: Unifying Language Learning Paradigms
• Speaker Normalization for Self-supervised Speech Emotion Recognition
• MetaICL: Learning to Learn In Context
• Finetuned Language Models Are Zero-Shot Learners
• Speaker Attentive Speech Emotion Recognition
• Language Models are Few-Shot Learners
• Multimodal Speech Emotion Recognition Using Audio and Text
• Attention Is All You Need

Related

• Personalized Adaptation with Pre-trained Speech Encoders for Continuous Emotion Recognition
• Improving Speech-based Emotion Recognition with Contextual Utterance Analysis and LLMs
• In-Context Examples Matter: Improving Emotion Recognition in Conversation with Instruction Tuning
• Revise, Reason, and Recognize: LLM-Based Emotion Recognition via Emotion-Specific Prompts and ASR Error Correction
• SELM: Enhancing Speech Emotion Recognition for Out-of-Domain Scenarios