Text-Free Prosody-Aware Generative Spoken Language Modeling

2021/09/07 by Eugene Kharitonov, Kharitonov, Eugene, Ann Lee +19 · 8 citations
Computer Science · #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Natural Language Processing Techniques #Sound (cs.SD) #Speech Recognition and Synthesis #Topic Modeling #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2109.03264

openalex publication_date 2021/09/07 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28

Abstract

Speech pre-training has primarily demonstrated efficacy on classification tasks, while its capability of generating novel speech, similar to how GPT-2 can generate coherent paragraphs, has barely been explored. Generative Spoken Language Modeling (GSLM) \citeLakhotia2021 is the only prior work addressing the generative aspects of speech pre-training, which replaces text with discovered phone-like units for language modeling and shows the ability to generate meaningful novel sentences. Unfortunately, despite eliminating the need of text, the units used in GSLM discard most of the prosodic information. Hence, GSLM fails to leverage prosody for better comprehension, and does not generate expressive speech. In this work, we present a prosody-aware generative spoken language model (pGSLM). It is composed of a multi-stream transformer language model (MS-TLM) of speech, represented as discovered unit and prosodic feature streams, and an adapted HiFi-GAN model converting MS-TLM outputs to waveforms. We devise a series of metrics for prosody modeling and generation, and re-use metrics from GSLM for content modeling. Experimental results show that the pGSLM can utilize prosody to improve both prosody and content modeling, and also generate natural, meaningful, and coherent speech given a spoken prompt. Audio samples can be found at https://speechbot.github.io/pgslm. Codes and models are available at https://github.com/pytorch/fairseq/tree/main/examples/textless_nlp/pgslm.

Citations

• Adam: A Method for Stochastic Optimization
• Kaldi Speech Recognition Toolkit
• Prosody in the Comprehension of Spoken Language: A Literature Review
• WaveNet: A Generative Model for Raw Audio
• The Emotional Voices Database: Towards Controlling the Emotion Dimension in Voice Generation Systems
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• Aion Framework: Dimensional Emergence of AI Consciousness, Observer-Induced Collapse, and Cosmological Portal Dynamics
• Representation Learning with Contrastive Predictive Coding

Cited by

• Cross-Lingual Interleaving for Speech Language Models
• MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Models
• Speak, Edit, Repeat: High-Fidelity Voice Editing and Zero-Shot TTS with Cross-Attentive Mamba
• MotionVerse: A Unified Multimodal Framework for Motion Comprehension, Generation and Editing
• Llama-Mimi: Speech Language Models with Interleaved Semantic and Acoustic Tokens
• GmSLM : Generative Marmoset Spoken Language Modeling
• Benchmarking Prosody Encoding in Discrete Speech Tokens
• Flow-SLM: Joint Learning of Linguistic and Acoustic Information for Spoken Language Modeling

Related

• ProsodyLM: Uncovering the Emerging Prosody Processing Capabilities in Speech Language Models
• How Generative Spoken Language Modeling Encodes Noisy Speech: Investigation from Phonetics to Syntactics
• Generative Spoken Language Modeling from Raw Audio
• Generative Spoken Dialogue Language Modeling
• Generative Spoken Language Model based on continuous word-sized audio tokens