E-chat: Emotion-sensitive Spoken Dialogue System with Large Language Models

2023/12/31 by Hongfei Xue, Xue, Hongfei, Yuhao Liang +10 · 4 citations
Computer Science · Psychology · #Audio and Speech Processing (eess.AS) #Emotion and Mood Recognition #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and dialogue systems #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2401.00475

openalex publication_date 2023/12/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This study focuses on emotion-sensitive spoken dialogue in human-machine speech interaction. With the advancement of Large Language Models (LLMs), dialogue systems can handle multimodal data, including audio. Recent models have enhanced the understanding of complex audio signals through the integration of various audio events. However, they are unable to generate appropriate responses based on emotional speech. To address this, we introduce the Emotional chat Model (E-chat), a novel spoken dialogue system capable of comprehending and responding to emotions conveyed from speech. This model leverages an emotion embedding extracted by a speech encoder, combined with LLMs, enabling it to respond according to different emotional contexts. Additionally, we introduce the E-chat200 dataset, designed explicitly for emotion-sensitive spoken dialogue. In various evaluation metrics, E-chat consistently outperforms baseline model, demonstrating its potential in emotional comprehension and human-machine interaction.

Cited by

• X-Talk: On the Underestimated Potential of Modular Speech-to-Speech Dialogue System
• Towards General Auditory Intelligence: Large Multimodal Models for Machine Listening and Speaking
• EchoMind: An Interrelated Multi-level Benchmark for Evaluating Empathetic Speech Language Models
• Dual Information Speech Language Models for Emotional Conversations

Related

• Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory
• EmoNews: A Spoken Dialogue System for Expressive News Conversations
• GatedxLSTM: A Multimodal Affective Computing Approach for Emotion Recognition in Conversations
• Leveraging Chain of Thought towards Empathetic Spoken Dialogue without Corresponding Question-Answering Data
• DialogueLLM: Context and Emotion Knowledge-Tuned Large Language Models for Emotion Recognition in Conversations