Generative Spoken Language Modeling from Raw Audio

2021/02/01 by Lakhotia, Kushal, Kharitonov, Evgeny, Hsu, Wei-Ning +8 · 38 citations
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2102.01192

Abstract

We introduce Generative Spoken Language Modeling, the task of learning the acoustic and linguistic characteristics of a language from raw audio (no text, no labels), and a set of metrics to automatically evaluate the learned representations at acoustic and linguistic levels for both encoding and generation. We set up baseline systems consisting of a discrete speech encoder (returning pseudo-text units), a generative language model (trained on pseudo-text), and a speech decoder (generating a waveform from pseudo-text) all trained without supervision and validate the proposed metrics with human evaluation. Across 3 speech encoders (CPC, wav2vec 2.0, HuBERT), we find that the number of discrete units (50, 100, or 200) matters in a task-dependent and encoder-dependent way, and that some combinations approach text-based systems.

Cited by

• Memory Efficient Audio Synthesis with Decoupled Temporal Depth Diffusion Transformers
• SpidR-Adapt: A Universal Speech Representation Model for Few-Shot Adaptation
• SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervision
• MauBERT: Universal Phonetic Inductive Biases for Few-Shot Acoustic Units Discovery
• MR-FlowDPO: Multi-Reward Direct Preference Optimization for Flow-Matching Text-to-Music Generation
• WhAM: Towards A Translative Model of Sperm Whale Vocalization
• Cross-Lingual Interleaving for Speech Language Models
• KidSpeak: A General Multi-purpose LLM for Kids' Speech Recognition and Screening
• AV-Dialog: Spoken Dialogue Models with Audio-Visual Input
• MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Models
• Towards General Auditory Intelligence: Large Multimodal Models for Machine Listening and Speaking
• emg2speech: synthesizing speech from electromyography using self-supervised speech models
• Are These Even Words? Quantifying the Gibberishness of Generative Speech Models
• Robust Distortion-Free Watermark for Autoregressive Audio Generation Models
• Data-Centric Lessons To Improve Speech-Language Pretraining
• LongCat-Audio-Codec: An Audio Tokenizer and Detokenizer Solution Designed for Speech Large Language Models
• Closing the Gap Between Text and Speech Understanding in LLMs
• Benchmarking Language Modeling for Lossless Compression of Full-Fidelity Audio
• Heptapod: Language Modeling on Visual Signals
• Latent Speech-Text Transformer
• Speak, Edit, Repeat: High-Fidelity Voice Editing and Zero-Shot TTS with Cross-Attentive Mamba
• MOSS-Speech: Towards True Speech-to-Speech Models Without Text Guidance
• Scaling Spoken Language Models with Syllabic Speech Tokenization
• Optimizing Speech Language Models for Acoustic Consistency
• Enhancing Speech Large Language Models through Reinforced Behavior Alignment
• Leveraging Audio-Visual Data to Reduce the Multilingual Gap in Self-Supervised Speech Models
• Advancing Speech Understanding in Speech-Aware Language Models with GRPO
• Think, Verbalize, then Speak: Bridging Complex Thoughts and Comprehensible Speech
• Llama-Mimi: Speech Language Models with Interleaved Semantic and Acoustic Tokens
• GmSLM : Generative Marmoset Spoken Language Modeling
• Continuous Audio Language Models
• An Empirical Analysis of Discrete Unit Representations in Speech Language Modeling Pre-training
• Group Relative Policy Optimization for Speech Recognition
• MixedG2P-T5: G2P-free Speech Synthesis for Mixed-script texts using Speech Self-Supervised Learning and Language Model
• Who Gets the Mic? Investigating Gender Bias in the Speaker Assignment of a Speech-LLM
• Representing Speech Through Autoregressive Prediction of Cochlear Tokens
• Benchmarking Prosody Encoding in Discrete Speech Tokens
• Flow-SLM: Joint Learning of Linguistic and Acoustic Information for Spoken Language Modeling

Related

• The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling
• How Generative Spoken Language Modeling Encodes Noisy Speech: Investigation from Phonetics to Syntactics
• Generative Spoken Language Model based on continuous word-sized audio tokens
• Generative Adversarial Training for Text-to-Speech Synthesis Based on Raw Phonetic Input and Explicit Prosody Modelling
• Augmentation Invariant Discrete Representation for Generative Spoken Language Modeling