Towards Bitrate-Efficient and Noise-Robust Speech Coding with Variable Bitrate RVQ

2025/06/19 by Chae, Yunkee, Lee, Kyogu · 1 citation
Computer Science · #Advanced Data Compression Techniques #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and Audio Processing #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2506.16538

openalex publication_date 2025/06/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Residual Vector Quantization (RVQ) has become a dominant approach in neural speech and audio coding, providing high-fidelity compression. However, speech coding presents additional challenges due to real-world noise, which degrades compression efficiency. Standard codecs allocate bits uniformly, wasting bitrate on noise components that do not contribute to intelligibility. This paper introduces a Variable Bitrate RVQ (VRVQ) framework for noise-robust speech coding, dynamically adjusting bitrate per frame to optimize rate-distortion trade-offs. Unlike constant bitrate (CBR) RVQ, our method prioritizes critical speech components while suppressing residual noise. Additionally, we integrate a feature denoiser to further improve noise robustness. Experimental results show that VRVQ improves rate-distortion trade-offs over conventional methods, achieving better compression efficiency and perceptual quality in noisy conditions. Samples are available at our project page: https://yoongi43.github.io/noise_robust_vrvq/.

Citations

• Variable Bitrate Residual Vector Quantization for Audio Coding
• Moshi: a speech-text foundation model for real-time dialogue
• EARS: An Anechoic Fullband Speech Dataset Benchmarked for Speech Enhancement and Dereverberation
• An Investigation of Incorporating Mamba for Speech Enhancement
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• High-Fidelity Audio Compression with Improved RVQGAN
• Selective compression learning of latent representations for variable-rate image compression
• High Fidelity Neural Audio Compression
• Optimizing Image Compression via Joint Learning with Denoising
• Disentangling speech from surroundings with neural embeddings
• End-to-End Neural Speech Coding for Real-Time Communications
• SoundStream: An End-to-End Neural Audio Codec
• UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation
• Enhancing into the codec: Noise Robust Speech Coding with Vector-Quantized Autoencoders
• HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
• Neural Networks Fail to Learn Periodic Functions and How to Fix It
• MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
• WHAM!: Extending Speech Separation to Noisy Environments
• Learning Content-Weighted Deep Image Compression
• SDR - half-baked or well done?
• Efficient Neural Audio Synthesis
• Conditional Probability Models for Deep Image Compression
• Neural Discrete Representation Learning
• FiLM: Visual Reasoning with a General Conditioning Layer
• Learning Convolutional Networks for Content-weighted Image Compression
• WaveNet: A Generative Model for Raw Audio
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• A Neural Speech Codec for Noise Robust Speech Coding

Cited by

• SUNAC: Source-aware Unified Neural Audio Codec

Related

• Variable Bitrate Residual Vector Quantization for Audio Coding
• ERVQ: Enhanced Residual Vector Quantization with Intra-and-Inter-Codebook Optimization for Neural Audio Codecs
• Improving Test-Time Performance of RVQ-based Neural Codecs
• Enhancing Noise Robustness for Neural Speech Codecs through Resource-Efficient Progressive Quantization Perturbation Simulation
• NDVQ: Robust Neural Audio Codec with Normal Distribution-Based Vector Quantization