DualCodec: A Low-Frame-Rate, Semantically-Enhanced Neural Audio Codec for Speech Generation

2025/05/19 by Li, Jiaqi, Lin, Xiaolong, Li, Zhekai +5 · 10 citations
#Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2505.13000

Abstract

Neural audio codecs form the foundational building blocks for language model (LM)-based speech generation. Typically, there is a trade-off between frame rate and audio quality. This study introduces a low-frame-rate, semantically enhanced codec model. Existing approaches distill semantically rich self-supervised (SSL) representations into the first-layer codec tokens. This work proposes DualCodec, a dual-stream encoding approach that integrates SSL and waveform representations within an end-to-end codec framework. In this setting, DualCodec enhances the semantic information in the first-layer codec and enables the codec system to maintain high audio quality while operating at a low frame rate. Note that a low-frame-rate codec improves the efficiency of speech generation. Experimental results on audio codec and speech generation tasks confirm the effectiveness of the proposed DualCodec compared to state-of-the-art codec systems, such as Mimi Codec, SpeechTokenizer, DAC, and Encodec. Demos are available at: https://dualcodec.github.io, code is available at: https://github.com/jiaqili3/DualCodec

Citations

• Recent Advances in Discrete Speech Tokens: A Review
• Overview of the Amphion Toolkit (v0.2)
• TS3-Codec: Transformer-Based Simple Streaming Single Codec
• Moshi: a speech-text foundation model for real-time dialogue
• Investigating Neural Audio Codecs for Speech Language Model-Based Speech Generation
• FireRedTTS: A Foundation Text-To-Speech Framework for Industry-Level Generative Speech Applications
• MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer
• Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model
• WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling
• CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens
• Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation
• Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generation
• VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers
• Seed-TTS: A Family of High-Quality Versatile Speech Generation Models
• VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild
• Seamless: Multilingual Expressive and Streaming Speech Translation
• RepCodec: A Speech Representation Codec for Speech Tokenization
• High-Fidelity Audio Compression with Improved RVQGAN
• Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis
• SoundStorm: Efficient Parallel Audio Generation
• HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec
• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
• Robust Speech Recognition via Large-Scale Weak Supervision
• High Fidelity Neural Audio Compression
• AudioLM: a Language Modeling Approach to Audio Generation
• UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022
• MaskGIT: Masked Generative Image Transformer
• A ConvNet for the 2020s
• A ConvNet for the 2020s
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training
• Layer-wise Analysis of a Self-supervised Speech Representation Model
• SoundStream: An End-to-End Neural Audio Codec
• HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
• Neural Networks Fail to Learn Periodic Functions and How to Fix It
• Neural Discrete Representation Learning
• Attention Is All You Need
• Deep Residual Learning for Image Recognition

Cited by

• JEPA as a Neural Tokenizer: Learning Robust Speech Representations with Density Adaptive Attention
• SpeechJudge: Towards Human-Level Judgment for Speech Naturalness
• U-Codec: Ultra Low Frame-rate Neural Speech Codec for Fast High-fidelity Speech Generation
• TASLA: Text-Aligned Speech Tokens with Multiple Layer-Aggregation
• FlexiCodec: A Dynamic Neural Audio Codec for Low Frame Rates
• Assessing speech quality metrics for evaluation of neural audio codecs under clean speech conditions
• AUV: Teaching Audio Universal Vector Quantization with Single Nested Codebook
• DeCodec: Rethinking Audio Codecs as Universal Disentangled Representation Learners
• TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling
• LM-SPT: LM-Aligned Semantic Distillation for Speech Tokenization
• Advancing Zero-shot Text-to-Speech Intelligibility across Diverse Domains via Preference Alignment

Related

• Low Frame-rate Speech Codec: a Codec Designed for Fast High-quality Speech LLM Training and Inference
• SAC: Neural Speech Codec with Semantic-Acoustic Dual-Stream Quantization
• Investigating Neural Audio Codecs for Speech Language Model-Based Speech Generation
• FlexiCodec: A Dynamic Neural Audio Codec for Low Frame Rates
• Optimizing Neural Speech Codec for Low-Bitrate Compression via Multi-Scale Encoding