SimulMEGA: MoE Routers are Advanced Policy Makers for Simultaneous Speech Translation

2025/09/01 by Chenyang Le, Bing Han, Le, Chenyang +7
Computer Science · #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Natural Language Processing Techniques #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and dialogue systems #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2509.01200

openalex publication_date 2025/09/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Simultaneous Speech Translation (SimulST) enables real-time cross-lingual communication by jointly optimizing speech recognition and machine translation under strict latency constraints. Existing systems struggle to balance translation quality, latency, and semantic coherence, particularly in multilingual many-to-many scenarios where divergent read and write policies hinder unified strategy learning. In this paper, we present SimulMEGA (Simultaneous Generation by Mixture-of-Experts Gating), an unsupervised policy learning framework that combines prefix-based training with a Mixture-of-Experts refiner to learn effective read and write decisions in an implicit manner, without adding inference-time overhead. Our design requires only minimal modifications to standard transformer architectures and generalizes across both speech-to-text and text-to-speech streaming tasks. Through comprehensive evaluation on six language pairs, our 500M parameter speech-to-text model outperforms the Seamless baseline, achieving under 7 percent BLEU degradation at 1.5 seconds average lag and under 3 percent at 3 seconds. We further demonstrate the versatility of SimulMEGA by extending it to streaming TTS with a unidirectional backbone, yielding superior latency quality tradeoffs.

Citations

• Large Language Models Are Read/Write Policy-Makers for Simultaneous Generation
• CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models
• PsFuture: A Pseudo-Future-based Zero-Shot Adaptive Policy for Simultaneous Machine Translation
• Textless Streaming Speech-to-Speech Translation using Semantic Speech Tokens
• Zero-Shot Text-to-Speech from Continuous Text Streams
• Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation
• Label-Synchronous Neural Transducer for E2E Simultaneous Speech Translation
• StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning
• LiveSpeech: Low-Latency Zero-shot Text-to-Speech via Autoregressive Modeling of Audio Discrete Codes
• Seed-TTS: A Family of High-Quality Versatile Speech Generation Models
• BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data
• Language Model is a Branch Predictor for Simultaneous Machine Translation
• Seamless: Multilingual Expressive and Streaming Speech Translation
• Efficient Monotonic Multihead Attention
• LibriSpeech-PC: Benchmark for Evaluation of Punctuation and Capitalization Capabilities of end-to-end ASR Models
• Speak While You Think: Streaming Speech Synthesis During Text Generation
• Learning Optimal Policy for Simultaneous Machine Translation via Binary Search
• Hidden Markov Transformer for Simultaneous Machine Translation
• Information-Transport-based Policy for Simultaneous Translation
• Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition
• FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech
• Large-Scale Streaming End-to-End Speech Translation with Neural Transducers
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognition
• Learning When to Translate for Streaming Speech
• LoRA: Low-Rank Adaptation of Large Language Models
• RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformer
• VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation
• SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translation
• SimulEval: An Evaluation Toolkit for Simultaneous Translation
• Common Voice: A Massively-Multilingual Speech Corpus
• Monotonic Multihead Attention
• Monotonic Infinite Lookback Attention for Simultaneous Machine Translation
• Online and Linear-Time Attention by Enforcing Monotonic Alignments
• SyncSpeech: Low-Latency and Efficient Dual-Stream Text-to-Speech based on Temporal Masked Transformer

Related

• Efficient and Adaptive Simultaneous Speech Translation with Fully Unidirectional Architecture
• StreamUni: Achieving Streaming Speech Translation with a Unified Large Speech-Language Model
• Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognition
• FASST: Fast LLM-based Simultaneous Speech Translation
• StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning