High-Fidelity Simultaneous Speech-To-Speech Translation

2025/02/05 by Tom Labiausse, Laurent Mazaré, Labiausse, Tom +10 · 13 voices · 7 citations
Computer Science · #Speech Recognition and Synthesis #cs.CL #cs.SD #eess.AS

paper · pdf · doi:10.48550/arxiv.2502.03382

openalex publication_date 2025/02/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce Hibiki, a decoder-only model for simultaneous speech translation. Hibiki leverages a multistream language model to synchronously process source and target speech, and jointly produces text and audio tokens to perform speech-to-text and speech-to-speech translation. We furthermore address the fundamental challenge of simultaneous interpretation, which unlike its consecutive counterpart, where one waits for the end of the source utterance to start translating, adapts its flow to accumulate just enough context to produce a correct translation in real-time, chunk by chunk. To do so, we introduce a weakly-supervised method that leverages the perplexity of an off-the-shelf text translation system to identify optimal delays on a per-word basis and create aligned synthetic data. After supervised training, Hibiki performs adaptive, simultaneous speech translation with vanilla temperature sampling. On a French-English simultaneous speech translation task, Hibiki demonstrates state-of-the-art performance in translation quality, speaker fidelity and naturalness. Moreover, the simplicity of its inference process makes it compatible with batched translation and even real-time on-device deployment. We provide examples as well as models and inference code.

Citations

• Attention Is All You Need

Cited by

• Simultaneous Speech-to-Speech Translation Without Aligned Data
• The seven roles of generative AI: Potential & pitfalls in combatting misinformation
• Can Speech LLMs Think while Listening?
• Llama-Mimi: Speech Language Models with Interleaved Semantic and Acoustic Tokens
• Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling
• Continuous Audio Language Models
• REINA: Regularized Entropy Information-Based Loss for Efficient Simultaneous Speech Translation

Discussions

• High-fidelity simultaneous speech-to-speech translation [hn, 115 points, 57 comments]
• Hibiki by kyutai, a simultaneous speech-to-speech translation model, currently supporting FR to EN [lemmy, 7 points, 1 comments]
• High-Fidelity Simultaneous Speech-to-Speech Translation [hn, 6 points, 1 comments]
• https://arxiv.org/abs/2502.03382 Hibikiという同時音声翻訳モデルが発表されました。 このモデルは、ソースとターゲットの音声を同時に処理し、テキストと音声トークンを生成します。 リアルタイムでの翻訳を可能にするため、遅延を最適化する弱教師あり学習法を導入しています。 [bsky, 1 points, 0 comments]
• High-Fidelity Simultaneous Speech-to-Speech Translation [bsky, 0 points, 0 comments]
• High-Fidelity Simultaneous Speech-to-Speech Translation #HackerNews https://arxiv.org/abs/2502.03382 [bsky, 0 points, 0 comments]
• High-Fidelity Simultaneous Speech-to-Speech Translation https://arxiv.org/abs/2502.03382 https://news.ycombinator.com/item?id=44458877 [bsky, 0 points, 0 comments]
• High-Fidelity Simultaneous Speech-To-Speech Translation #machinetranslation #interpretation [bsky, 0 points, 0 comments]
• High-Fidelity Simultaneous Speech-to-Speech Translation https://arxiv.org/abs/2502.03382 [bsky, 0 points, 0 comments]
• ⚡ Hackernews Top story: High-Fidelity Simultaneous Speech-to-Speech Translation [bsky, 0 points, 0 comments]
• High-Fidelity Simultaneous Speech-to-Speech Translation https://arxiv.org/abs/2502.03382 (https://news.ycombinator.com/item?id=44458877) [bsky, 0 points, 0 comments]
• High-Fidelity Simultaneous Speech-to-Speech Translation https://arxiv.org/abs/2502.03382 (https://news.ycombinator.com/item?id=44458877) [bsky, 0 points, 0 comments]
• https://bsky.app/profile/buzzing.cc.web.brid.gy/post/3lt5btpvtbxw2 [bsky, 0 points, 0 comments]

Related

• Simultaneous Speech-to-Speech Translation Without Aligned Data
• An Empirical Study of End-to-end Simultaneous Speech Translation Decoding Strategies
• Re-Translation Strategies For Long Form, Simultaneous, Spoken Language\n Translation
• Seed LiveInterpret 2.0: End-to-end Simultaneous Speech-to-speech Translation with Your Voice
• FASST: Fast LLM-based Simultaneous Speech Translation