MSR-Codec: A Low-Bitrate Multi-Stream Residual Codec for High-Fidelity Speech Generation with Information Disentanglement

2025/09/16 by Jingyu Li, G Zhang, Li, Jingyu +5
Computer Science · #Advanced Data Compression Techniques #Audio and Speech Processing (eess.AS) #FOS: Electrical engineering #Speech Recognition and Synthesis #Speech and Audio Processing #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2509.13068

openalex publication_date 2025/09/16 · openalex created_date 2025/10/17 · openalex updated_date 2026/07/28

Abstract

Audio codecs are a critical component of modern speech generation systems. This paper introduces a low-bitrate, multi-scale residual codec that encodes speech into four distinct streams: semantic, timbre, prosody, and residual. This architecture achieves high-fidelity speech reconstruction at competitive low bitrates while demonstrating an inherent ability for information disentanglement. We construct a two-stage language model for text-to-speech (TTS) synthesis using this codec, which, despite its lightweight design and minimal data requirements, achieves a state-of-the-art Word Error Rate (WER) and superior speaker similarity compared to several larger models. Furthermore, the codec's design proves highly effective for voice conversion, enabling independent manipulation of speaker timbre and prosody. Our inference code, pre-trained models, and audio samples are available at https://github.com/herbertLJY/MSRCodec.

Citations

• Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis
• CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models
• Zero-shot Voice Conversion with Diffusion Transformers
• LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codec
• FireRedTTS: A Foundation Text-To-Speech Framework for Industry-Level Generative Speech Applications
• CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens
• Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generation
• VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers
• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models
• Fewer-token Neural Speech Codec with Time-invariant Codes
• CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Masking
• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
• High Fidelity Neural Audio Compression
• UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022
• Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verification
• WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognition
• SoundStream: An End-to-End Neural Audio Codec
• HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
• Fre-GAN: Adversarial Frequency-consistent Audio Synthesis
• CN-Celeb: multi-genre speaker recognition
• SEANet: A Multi-modal Speech Enhancement Network
• Voxceleb: Large-scale speaker verification in the wild
• LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech
• AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scale
• AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline
• Attention Is All You Need

Related

• Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generation
• MBCodec:Thorough disentangle for high-fidelity audio compression
• DisCo-Speech: Controllable Zero-Shot Speech Generation with A Disentangled Speech Codec
• LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codec
• MARS6: A Small and Robust Hierarchical-Codec Text-to-Speech Model