FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs

2024/07/04 by An, Keyu, Chen, Qian, Deng, Chong +30 · 30 citations
Computer Science · #Artificial Intelligence (cs.AI) #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #Speech and dialogue systems #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2407.04051

openalex publication_date 2024/07/04 · openalex created_date 2024/07/09 · openalex updated_date 2026/07/28

Abstract

This report introduces FunAudioLLM, a model family designed to enhance natural voice interactions between humans and large language models (LLMs). At its core are two innovative models: SenseVoice, which handles multilingual speech recognition, emotion recognition, and audio event detection; and CosyVoice, which facilitates natural speech generation with control over multiple languages, timbre, speaking style, and speaker identity. SenseVoice-Small delivers exceptionally low-latency ASR for 5 languages, and SenseVoice-Large supports high-precision ASR for over 50 languages, while CosyVoice excels in multi-lingual voice generation, zero-shot in-context learning, cross-lingual voice cloning, and instruction-following capabilities. The models related to SenseVoice and CosyVoice have been open-sourced on Modelscope and Huggingface, along with the corresponding training, inference, and fine-tuning codes released on GitHub. By integrating these models with LLMs, FunAudioLLM enables applications such as speech-to-speech translation, emotional voice chat, interactive podcasts, and expressive audiobook narration, thereby pushing the boundaries of voice interaction technology. Demos are available at https://fun-audio-llm.github.io, and the code can be accessed at https://github.com/FunAudioLLM.

Cited by

• Let Me Look at You: Advanced Facial Expression Modeling for Conversational Speech Synthesis
• Marco-ASR: A Principled and Metric-Driven Framework for Fine-Tuning Large-Scale ASR Models for Domain Adaptation
• Rare Word Recognition and Translation Without Fine-Tuning via Task Vector in Speech Models
• Foundation Model-based Evaluation of Neuropsychiatric Disorders: A Lifespan-Inclusive, Multi-Modal, and Multi-Lingual Study
• X-Talk: On the Underestimated Potential of Modular Speech-to-Speech Dialogue System
• Large Speech Model Enabled Semantic Communication
• PFEA: An LLM-based High-Level Natural Language Planning and Feedback Embodied Agent for Human-Centered AI
• UltraVoice: Scaling Fine-Grained Style-Controlled Speech Conversations for Spoken Dialogue Models
• LSZone: A Lightweight Spatial Information Modeling Architecture for Real-time In-car Multi-zone Speech Separation
• AV-EMO-Reasoning: Benchmarking Emotional Reasoning Capabilities in Omni-modal LLMS with Audio-visual Cues
• FlexiCodec: A Dynamic Neural Audio Codec for Low Frame Rates
• Plug-and-Play Emotion Graphs for Compositional Prompting in Zero-Shot Speech Emotion Recognition
• Word-Level Emotional Expression Control in Zero-Shot Text-to-Speech Synthesis
• HiKE: Hierarchical Evaluation Framework for Korean-English Code-Switching Speech Recognition
• SenSE: Semantic-Aware High-Fidelity Universal Speech Enhancement
• Code-switching Speech Recognition Under the Lens: Model- and Data-Centric Perspectives
• WEST: LLM based Speech Toolkit for Speech Understanding, Generation, and Interaction
• PART: Progressive Alignment Representation Training for Multilingual Speech-To-Text with LLMs
• HarmoniFuse: A Component-Selective and Prompt-Adaptive Framework for Multi-Task Speech Language Modeling
• WenetSpeech-Chuan: A Large-Scale Sichuanese Corpus with Rich Annotation for Dialectal Speech Processing
• EMO-Reasoning: Benchmarking Emotional Reasoning Capabilities in Spoken Dialogue Systems
• MNV-17: A High-Quality Performative Mandarin Dataset for Nonverbal Vocalization Recognition in Speech
• SynParaSpeech: Automated Synthesis of Paralinguistic Datasets for Speech Generation and Understanding
• MELA-TTS: Joint transformer-diffusion model with representation alignment for speech synthesis
• Fun-ASR Technical Report
• CabinSep: IR-Augmented Mask-Based MVDR for Real-Time In-Car Speech Separation with Distributed Heterogeneous Arrays
• Empathy Omni: Enabling Empathetic Speech Response Generation through Large Language Models
• M³PDB: A Multimodal, Multi-Label, Multilingual Prompt Database for Speech Generation
• UniTalker: Conversational Speech-Visual Synthesis
• NVSpeech: An Integrated and Scalable Pipeline for Human-Like Speech Modeling with Paralinguistic Vocalizations

Related

• Recent Advances in Speech Language Models: A Survey
• AudioPaLM: A Large Language Model That Can Speak and Listen
• Voila: Voice-Language Foundation Models for Real-Time Autonomous Interaction and Voice Role-Play
• Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities
• VoiceBench: Benchmarking LLM-Based Voice Assistants