Auden-Voice: General-Purpose Voice Encoder for Speech and Language Understanding

2025/11/19 by Huo, Mingyue, Tseng, Wei-Cheng, Shao, Yiwen +2 · 1 citation
Computer Science · Psychology · #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Phonetics and Phonology Research #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and Audio Processing #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2511.15145

openalex publication_date 2025/11/19 · openalex created_date 2025/11/23 · openalex updated_date 2026/07/28

Abstract

Human voice encodes both identity and paralinguistic cues, yet encoders in large audio-language models (LALMs) rarely balance both aspects. In this work, we present a study toward building a general-purpose voice encoder that captures nuanced voice cues. Through a comprehensive evaluation, we find that multi-task training yields the most balanced representations, whereas contrastive language-audio pretraining (CLAP) primarily improves retrieval without enhancing paralinguistic understanding. Our final encoder, Auden-Voice, also demonstrates strong performance when integrated with LLMs. The code and training recipes will be released with the audio understanding toolkit Auden.

Citations

• Efficient Scaling for LLM-based ASR
• Step-Audio 2 Technical Report
• RA-CLAP: Relation-Augmented Emotional Speaking Style Contrastive Language-Audio Pretraining For Speech Retrieval
• Scaling Rich Style-Prompted Text-to-Speech Datasets
• Beyond Speaker Identity: Text Guided Target Speech Extraction
• MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark
• Overview of Speaker Modeling and Its Applications: From the Lens of Deep Speaker Representation Learning
• Qwen2-Audio Technical Report
• Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation
• AudioBench: A Universal Benchmark for Audio Large Language Models
• ParaCLAP -- Towards a general language-audio model for computational paralinguistic tasks
• EARS: An Anechoic Fullband Speech Dataset Benchmarked for Speech Enhancement and Dereverberation
• An Embarrassingly Simple Approach for LLM with Strong ASR Capacity
• AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension
• Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities
• emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation
• SALMONN: Towards Generic Hearing Abilities for Large Language Models
• Zipformer: A faster and better encoder for automatic speech recognition
• Prompting Audios Using Acoustic Properties For Emotion Representation
• EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis
• Vesper: A Compact and Effective Pretrained Model for Speech Emotion Recognition
• BEATs: Audio Pre-Training with Acoustic Tokenizers
• Robust Speech Recognition via Large-Scale Weak Supervision
• Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkit
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• x-vectors meet emotions: A study on dependencies between emotion and\n speaker recognition
• Common Voice: A Massively-Multilingual Speech Corpus
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in\n Conversations
• Generalized End-to-End Loss for Speaker Verification

Cited by

• Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation

Related

• AudioPaLM: A Large Language Model That Can Speak and Listen
• Competitive Audio-Language Models with Data-Efficient Single-Stage Training on Public Data
• FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs
• DSCLAP: Domain-Specific Contrastive Language-Audio Pre-Training
• LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT