Speech-to-Text Translation with Phoneme-Augmented CoT: Enhancing Cross-Lingual Transfer in Low-Resource Scenarios

2025/05/30 by Gállego, Gerard I., Pareras, Oriol, Garcia, Martí Cortada +2 · 3 citations
#Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2505.24691

Abstract

We propose a Speech-to-Text Translation (S2TT) approach that integrates phoneme representations into a Chain-of-Thought (CoT) framework to improve translation in low-resource and zero-resource settings. By introducing phoneme recognition as an intermediate step, we enhance cross-lingual transfer, enabling translation even for languages with no labeled speech data. Our system builds on a multilingual LLM, which we extend to process speech and phonemes. Training follows a curriculum learning strategy that progressively introduces more complex tasks. Experiments on multilingual S2TT benchmarks show that phoneme-augmented CoT improves translation quality in low-resource conditions and enables zero-resource translation, while slightly impacting high-resource performance. Despite this trade-off, our findings demonstrate that phoneme-based CoT is a promising step toward making S2TT more accessible across diverse languages.

Cited by

• Listening or Reading? Evaluating Speech Awareness in Chain-of-Thought Speech-to-Text Translation
• POWSM: A Phonetic Open Whisper-Style Speech Foundation Model
• Can Speech LLMs Think while Listening?

Related

• Using Phonemes in cascaded S2S translation pipeline
• Revisiting Direct Speech-to-Text Translation with Speech LLMs: Better Scaling than CoT Prompting?
• Improving Cross-Lingual Transfer Learning for End-to-End Speech Recognition with Speech Translation
• AlloST: Low-resource Speech Translation without Source Transcription
• Low-Resource Speech-to-Text Translation