Optimizing Speech Language Models for Acoustic Consistency

2025/09/30 by Rohanian, Morteza, Krauthammer, Michael
#Computation and Language (cs.CL) #FOS: Computer and information sciences #Sound (cs.SD)

paper · doi:10.48550/arxiv.2509.26276

Abstract

We study speech language models that incorporate semantic initialization and planning losses to achieve robust and consistent generation. Our approach initializes speech tokens with self-supervised features, applies a light alignment loss, and trains with thinning and auxiliary objectives that target robustness and content planning. We train three models: a 0.7B speech-only model, a 1.0B speech-only model, and a 1.0B interleaved model with both text and speech. Acoustic studies show that the speech-only models achieve the highest consistency across speaker, gender, sentiment, room, and background factors, surpassing larger systems. Interleaving improves lexical and syntactic probes and semantic--acoustic alignment but reduces consistency. Linear probes show that our initialization biases the model toward content structure while trading off prosody detail. These results show that LM-side design and training mix control the balance between acoustic stability and semantic grounding without changes to the tokenizer or runtime architecture. A demo and model weights are available for exploration.

Citations

• Flow-SLM: Joint Learning of Linguistic and Acoustic Information for Spoken Language Modeling
• Efficient Interleaved Speech Modeling through Knowledge Distillation
• Moshi: a speech-text foundation model for real-time dialogue
• Salmon: A Suite for Acoustic Language Model Evaluation
• LAST: Language Model Aware Speech Tokenization
• WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling
• Spirit LM: Interleaved Spoken and Written Language Model
• Textually Pretrained Speech Language Models
• SoundStorm: Efficient Parallel Audio Generation
• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
• AudioLM: a Language Modeling Approach to Audio Generation
• The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usage
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
• Generative Spoken Language Modeling from Raw Audio

Related

• Llama-Mimi: Speech Language Models with Interleaved Semantic and Acoustic Tokens
• GenSE: Generative Speech Enhancement via Language Models using Hierarchical Modeling
• SpeechAlign: Aligning Speech Generation to Human Preferences
• LAST: Language Model Aware Speech Tokenization
• Content is What Remains: Invariant Speech Tokenization from Parallel Utterances