Segmentation-Variant Codebooks for Preservation of Paralinguistic and Prosodic Information

2025/05/21 by Nicholas Sanders, Yuanchao Li, Sanders, Nicholas +5
Computer Science · #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Natural Language Processing Techniques #Sound (cs.SD) #Speech Recognition and Synthesis #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2505.15667

openalex publication_date 2025/05/21 · openalex created_date 2025/10/19 · openalex updated_date 2026/08/01

Abstract

Quantization in SSL speech models (e.g., HuBERT) improves compression and performance in tasks like language modeling, resynthesis, and text-to-speech but often discards prosodic and paralinguistic information (e.g., emotion, prominence). While increasing codebook size mitigates some loss, it inefficiently raises bitrates. We propose Segmentation-Variant Codebooks (SVCs), which quantize speech at distinct linguistic units (frame, phone, word, utterance), factorizing it into multiple streams of segment-specific discrete features. Our results show that SVCs are significantly more effective at preserving prosodic and paralinguistic information across probing tasks. Additionally, we find that pooling before rather than after discretization better retains segment-level information. Resynthesis experiments further confirm improved style realization and slightly improved quality while preserving intelligibility.

Citations

• Text-to-Speech for Unseen Speakers via Low-Complexity Discrete Unit-Based Frame Selection
• A layer-wise analysis of Mandarin and English suprasegmentals in SSL speech models
• EMO-Codec: An In-Depth Look at Emotion Preservation capacity of Legacy and Neural Codec Models With Subjective and Objective Evaluations
• DiscreteSLU: A Large Language Model with Self-Supervised Discrete Speech Units for Spoken Language Understanding
• The Interspeech 2024 Challenge on Speech Processing Using Discrete Units
• Crossmodal ASR Error Correction with Discrete Speech Units
• Layer-Wise Analysis of Self-Supervised Acoustic Word Embeddings: A Study on Speech Emotion Recognition
• EmphAssess : a Prosodic Benchmark on Assessing Emphasis Transfer in Speech-to-Speech Models
• Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Study
• EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis
• On the Use of Self-Supervised Speech Representations in Spontaneous Speech Synthesis
• Speech Emotion Diarization: Which Emotion Appears When?
• Analyzing Acoustic Word Embeddings from Pre-trained Self-supervised Speech Models
• High Fidelity Neural Audio Compression
• Exploration of A Self-Supervised Speech Model: A Study on Emotional Corpora
• UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022
• A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understanding
• Exploring Wav2vec 2.0 fine-tuning for improved speech emotion recognition
• SoundStream: An End-to-End Neural Audio Codec
• HuBERT: Self-Supervised Speech Representation Learning by Masked\n Prediction of Hidden Units
• SUPERB: Speech processing Universal PERformance Benchmark
• HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
• Neural Discrete Representation Learning

Related

• SecoustiCodec: Cross-Modal Aligned Streaming Single-Codecbook Speech Codec
• SECodec: Structural Entropy-based Compressive Speech Representation Codec for Speech Language Models
• Revisiting speech segmentation and lexicon learning with better features
• Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generation
• Content is What Remains: Invariant Speech Tokenization from Parallel Utterances