SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing

2021/10/14 by Junyi Ao, Rui Wang, Ao, Junyi +25 · 37 citations
Computer Science · Engineering · #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Natural Language Processing Techniques #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and Audio Processing #cs.CL #cs.LG #cs.SD #eess.AS #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2110.07205

Accepted by ACL 2022 main conference

openalex publication_date 2021/10/14 · arxiv created 2022/05/24 · arxiv updated 2022/05/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Motivated by the success of T5 (Text-To-Text Transfer Transformer) in pre-trained natural language processing models, we propose a unified-modal SpeechT5 framework that explores the encoder-decoder pre-training for self-supervised speech/text representation learning. The SpeechT5 framework consists of a shared encoder-decoder network and six modal-specific (speech/text) pre/post-nets. After preprocessing the input speech/text through the pre-nets, the shared encoder-decoder network models the sequence-to-sequence transformation, and then the post-nets generate the output in the speech/text modality based on the output of the decoder. Leveraging large-scale unlabeled speech and text data, we pre-train SpeechT5 to learn a unified-modal representation, hoping to improve the modeling capability for both speech and text. To align the textual and speech information into this unified semantic space, we propose a cross-modal vector quantization approach that randomly mixes up speech/text states with latent units as the interface between encoder and decoder. Extensive evaluations show the superiority of the proposed SpeechT5 framework on a wide variety of spoken language processing tasks, including automatic speech recognition, speech synthesis, speech translation, voice conversion, speech enhancement, and speaker identification. We release our code and model at https://github.com/microsoft/SpeechT5.

Cited by

• T5Gemma 2: Seeing, Reading, and Understanding Longer
• The Affective Bridge: Preserving Speech Representations while Enhancing Deepfake Detection vian emotional Constraints
• TriDF: Evaluating Perception, Detection, and Hallucination for Interpretable DeepFake Detection
• Bayesian Speech synthesizers Can Learn from Multiple Teachers
• Edge-Based Speech Transcription and Synthesis for Kinyarwanda and Swahili Languages
• DeepfakeBench-MM: A Comprehensive Benchmark for Multimodal Deepfake Detection
• Can Current Detectors Catch Face-to-Voice Deepfake Attacks?
• EchoFake: A Replay-Aware Dataset for Practical Speech Deepfake Detection
• NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matching
• UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Models
• Towards Unsupervised Speech Recognition at the Syllable-Level
• When Voice Matters: Evidence of Gender Disparity in Positional Bias of SpeechLLMs
• From Text to Talk: Audio-Language Model Needs Non-Autoregressive Joint Training
• HarmoniFuse: A Component-Selective and Prompt-Adaptive Framework for Multi-Task Speech Language Modeling
• LadderSym: A Multimodal Interleaved Transformer for Music Practice Error Detection
• Semantic-Enhanced Cross-Modal Place Recognition for Robust Robot Localization
• ERIS: Evolutionary Real-world Interference Scheme for Jailbreaking Audio Large Models
• WhisTLE: Deeply Supervised, Text-Only Domain Adaptation for Pretrained Speech Recognition Transformers
• MixedG2P-T5: G2P-free Speech Synthesis for Mixed-script texts using Speech Self-Supervised Learning and Language Model
• CLEAR: Continuous Latent Autoregressive Modeling for High-quality and Low-latency Speech Synthesis
• MiDashengLM: Efficient Audio Understanding with General Audio Captions
• MMBERT: Scaled Mixture-of-Experts Multimodal BERT for Robust Chinese Hate Speech Detection under Cloaking Perturbations
• Model-free Speculative Decoding for Transformer-based ASR with Token Map Drafting
• Weak Supervision Techniques towards Enhanced ASR Models in Industry-level CRM Systems
• A Technique for Isolating Lexically-Independent Phonetic Dependencies in Generative CNNs
• TTSDS2: Resources and Benchmark for Evaluating Human-Quality Text to Speech Systems
• TESU-LLM: Training Speech-LLMs Without Speech via Unified Encoder Alignment
• From Words to Waves: Analyzing Concept Formation in Speech and Text-Based Foundation Models
• BaryIR: Learning Multi-Source Unified Representation in Continuous Barycenter Space for Generalizable All-in-One Image Restoration
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• JALMBench: Benchmarking Jailbreak Vulnerabilities in Audio Language Models
• Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Models
• Speechless: Speech Instruction Training Without Speech for Low Resource Languages
• UDDETTS: Unifying Discrete and Dimensional Emotions for Controllable Emotional Text-to-Speech
• Multi-Modal Explainable Medical AI Assistant for Trustworthy Human-AI Collaboration
• ClonEval: An Open Voice Cloning Benchmark
• Text-Utilization for Encoder-dominated Speech Recognition Models

Related

• SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Data
• SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training
• Unified Speech-Text Pre-training for Speech Translation and Recognition
• token2vec: A Joint Self-Supervised Pre-training Framework Using Unpaired Speech and Text
• VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation