Leveraging Unit Language Guidance to Advance Speech Modeling in Textless Speech-to-Speech Translation

2025/05/21 by Yuhao Zhang, Xiangnan Ma, Zhang, Yuhao +17 · 1 citation
Computer Science · #Artificial Intelligence (cs.AI) #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Natural Language Processing Techniques #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and dialogue systems #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2505.15333

openalex publication_date 2025/05/21 · openalex created_date 2025/10/19 · openalex updated_date 2026/07/28

Abstract

The success of building textless speech-to-speech translation (S2ST) models has attracted much attention. However, S2ST still faces two main challenges: 1) extracting linguistic features for various speech signals, called cross-modal (CM), and 2) learning alignment of difference languages in long sequences, called cross-lingual (CL). We propose the unit language to overcome the two modeling challenges. The unit language can be considered a text-like representation format, constructed using n-gram language modeling. We implement multi-task learning to utilize the unit language in guiding the speech modeling process. Our initial results reveal a conflict when applying source and target unit languages simultaneously. We propose task prompt modeling to mitigate this conflict. We conduct experiments on four languages of the Voxpupil dataset. Our method demonstrates significant improvements over a strong baseline and achieves performance comparable to models trained with text.

Citations

• Rethinking and Improving Multi-task Learning for End-to-end Speech Translation
• Acoustic BPE for Speech Generation with Discrete Tokens
• DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translation
• UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units
• Textless Direct Speech-to-Speech Translation with Discrete Speech Representation
• Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languages
• Textless Speech-to-Speech Translation on Real Data
• Translatotron 2: High-quality direct speech-to-speech translation with voice preservation
• Direct speech-to-speech translation with discrete units
• HuBERT: Self-Supervised Speech Representation Learning by Masked\n Prediction of Hidden Units
• Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders
• Speech Resynthesis from Discrete Disentangled Self-Supervised Representations
• VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation
• HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
• UWSpeech: Speech to Speech Translation for Unwritten Languages
• Speech-to-speech Translation between Untranscribed Unknown Languages
• Direct speech-to-speech translation with a sequence-to-sequence model
• SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing
• Attention Is All You Need

Cited by

• EchoX: Towards Mitigating Acoustic-Semantic Gap via Echo Training for Speech-to-Speech LLMs

Related

• Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translation
• Direct speech-to-speech translation with discrete units
• Toward Joint Language Modeling for Speech Units and Text
• Analyzing Speech Unit Selection for Textless Speech-to-Speech Translation
• ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Text Translation