Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling

2025/09/10 by Neil Zeghidour, Eugene Kharitonov, Zeghidour, Neil +15 · 5 citations
Computer Science · Engineering · #Computation and Language (cs.CL) #Data Stream Mining Techniques #FOS: Computer and information sciences #Fault Detection and Control Systems #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2509.08753

openalex publication_date 2025/09/10 · openalex created_date 2025/10/19 · openalex updated_date 2026/07/28

Abstract

We introduce Delayed Streams Modeling (DSM), a flexible formulation for streaming, multimodal sequence-to-sequence learning. Sequence-to-sequence generation is often cast in an offline manner, where the model consumes the complete input sequence before generating the first output timestep. Alternatively, streaming sequence-to-sequence rely on learning a policy for choosing when to advance on the input stream, or write to the output stream. DSM instead models already time-aligned streams with a decoder-only language model. By moving the alignment to a pre-processing step,and introducing appropriate delays between streams, DSM provides streaming inference of arbitrary output sequences, from any input combination, making it applicable to many sequence-to-sequence problems. In particular, given text and audio streams, automatic speech recognition (ASR) corresponds to the text stream being delayed, while the opposite gives a text-to-speech (TTS) model. We perform extensive experiments for these two major sequence-to-sequence tasks, showing that DSM provides state-of-the-art performance and latency while supporting arbitrary long sequences, being even competitive with offline baselines. Code, samples and demos are available at https://github.com/kyutai-labs/delayed-streams-modeling

Citations

• CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training
• Training and Inference Efficiency of Encoder-Decoder Speech Models
• Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs
• High-Fidelity Simultaneous Speech-To-Speech Translation
• F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching
• Moshi: a speech-text foundation model for real-time dialogue
• CrisperWhisper: Accurate Timestamps on Verbatim Speech Transcriptions
• PaliGemma: A versatile 3B VLM for transfer
• Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation
• Streaming Decoder-Only Automatic Speech Recognition with Discrete Speech Units: A Pilot Study
• Decoder-only Streaming Transformer for Simultaneous Translation
• Seed-TTS: A Family of High-Quality Versatile Speech Generation Models
• CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations
• Proactive Detection of Voice Cloning with Localized Watermarking
• Seamless: Multilingual Expressive and Streaming Speech Translation
• Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling
• Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning
• Turning Whisper into Real-Time Transcription System
• AudioPaLM: A Large Language Model That Can Speak and Listen
• CML-TTS A Multilingual Dataset for Speech Synthesis in Low-Resource Languages
• Efficient Sequence Transduction by Jointly Predicting Tokens and Durations
• Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision
• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
• Robust Speech Recognition via Large-Scale Weak Supervision
• A Weakly-Supervised Streaming Multilingual Speech Model with Truly Zero-Shot Capability
• High Fidelity Neural Audio Compression
• AudioGen: Textually Guided Audio Generation
• Classifier-Free Diffusion Guidance
• Flamingo: a Visual Language Model for Few-Shot Learning
• Earnings-22: A Practical Benchmark for Accents in the Wild
• Autoregressive Image Generation using Residual Quantization
• SoundStream: An End-to-End Neural Audio Codec
• Direct Simultaneous Speech-to-Text Translation Assisted by Synchronized Streaming ASR
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• SPGISpeech: 5,000 hours of transcribed financial audio for fully\n formatted end-to-end speech recognition
• Zero-Shot Text-to-Image Generation
• VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation
• Taming Transformers for High-Resolution Image Synthesis
• A Better and Faster End-to-End Model for Streaming ASR
• Streaming Simultaneous Speech Translation with Augmented Memory Transformer
• Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss
• A comparison of end-to-end models for long-form speech recognition
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• Billion-scale semi-supervised learning for image classification
• Speech Model Pre-training for End-to-End Spoken Language Understanding
• Monotonic Chunkwise Attention
• Attention Is All You Need
• Online and Linear-Time Attention by Enforcing Monotonic Alignments
• Tacotron: Towards End-to-End Speech Synthesis
• Listen, Attend and Spell
• Sequence to Sequence Learning with Neural Networks
• Neural Machine Translation by Jointly Learning to Align and Translate
• Speech Recognition with Deep Recurrent Neural Networks
• Sequence Transduction with Recurrent Neural Networks
• Efficient Bayesian Inference for Generalized Bradley-Terry Models
• Long Short-Term Memory
• RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS

Cited by

• Cross-Lingual Interleaving for Speech Language Models
• Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluation
• Game-Time: Evaluating Temporal Dynamics in Spoken Language Models
• KAME: Tandem Architecture for Enhancing Knowledge in Real-Time Speech-to-Speech Conversational AI
• PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models
• Continuous Audio Language Models

Related

• Minimum Latency Training Strategies for Streaming Sequence-to-Sequence ASR
• High Performance Sequence-to-Sequence Model for Streaming Speech Recognition
• Dynamic latency speech recognition with asynchronous revision
• Improving Streaming Automatic Speech Recognition With Non-Streaming Model Distillation On Unsupervised Data
• Zero-Shot Text-to-Speech from Continuous Text Streams