Codec2Vec: Self-Supervised Speech Representation Learning Using Neural Speech Codecs

2025/11/20 by Wei‐Cheng Tseng, Tseng, Wei-Cheng, David Harwath +1 · 1 citation
Computer Science · Medicine · #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Speech Recognition and Synthesis #Speech and Audio Processing #Voice and Speech Disorders #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2511.16639

openalex publication_date 2025/11/20 · openalex created_date 2025/11/23 · openalex updated_date 2026/07/28

Abstract

Recent advancements in neural audio codecs have not only enabled superior audio compression but also enhanced speech synthesis techniques. Researchers are now exploring their potential as universal acoustic feature extractors for a broader range of speech processing tasks. Building on this trend, we introduce Codec2Vec, the first speech representation learning framework that relies exclusively on discrete audio codec units. This approach offers several advantages, including improved data storage and transmission efficiency, faster training, and enhanced data privacy. We explore masked prediction with various training target derivation strategies to thoroughly understand the effectiveness of this framework. Evaluated on the SUPERB benchmark, Codec2Vec achieves competitive performance compared to continuous-input models while reducing storage requirements by up to 16.5x and training time by 2.3x, showcasing its scalability and efficiency.

Citations

• Probing the Robustness Properties of Neural Speech Codecs
• DASB - Discrete Audio and Speech Benchmark
• Sustainable self-supervised learning for speech representations
• A Large-Scale Evaluation of Speech Foundation Models
• VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild
• Codec-SUPERB: An In-Depth Analysis of Sound Codec Models
• AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling
• The Faiss library
• Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Study
• Discrete Audio Representation as an Alternative to Mel-Spectrograms for Speaker and Speech Recognition
• EnCodecMAE: Leveraging neural codecs for universal audio representation learning
• SpeechX: Neural Codec Language Model as a Versatile Speech Transformer
• Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale
• High-Fidelity Audio Compression with Improved RVQGAN
• Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Compute
• VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation
• DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learning
• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
• MelHuBERT: A simplified HuBERT on Mel spectrograms
• High Fidelity Neural Audio Compression
• CoBERT: Self-Supervised Speech Representation Learning Through Code Representation Learning
• Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognition
• SoundStream: An End-to-End Neural Audio Codec
• HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
• SUPERB: Speech processing Universal PERformance Benchmark
• DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• Effectiveness of self-supervised pre-training for speech recognition
• Representation Learning with Contrastive Predictive Coding

Cited by

• Dissecting Sensitivity to Training Language in Self-Supervised Speech Learning Using Neural Audio Codec Tokens

Related

• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• Wav2vec-C: A Self-supervised Model for Speech Representation Learning
• wav2vec: Unsupervised Pre-training for Speech Recognition
• DualCodec: A Low-Frame-Rate, Semantically-Enhanced Neural Audio Codec for Speech Generation
• MBCodec:Thorough disentangle for high-fidelity audio compression