Instituto de Telecomunicações at IWSLT 2025: Aligning Small-Scale Speech and Language Models for Speech-to-Text Learning

2025/06/20 by Attanasio, Giuseppe, Sannigrahi, Sonal, Peters, Ben +1
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech Recognition and Synthesis #Speech and dialogue systems

paper · pdf · doi:10.48550/arxiv.2506.17019

openalex publication_date 2025/06/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper presents the IT-IST submission to the IWSLT 2025 Shared Task on Instruction Following Speech Processing. We submit results for the Short Track, i.e., speech recognition, translation, and spoken question answering. Our model is a unified speech-to-text model that integrates a pre-trained continuous speech encoder and text decoder through a first phase of modality alignment and a second phase of instruction fine-tuning. Crucially, we focus on using small-scale language model backbones (< 2B) and restrict to high-quality, CC-BY data along with synthetic data generation to supplement existing resources.

Citations

• From TOWER to SPIRE: Adding the Speech Modality to a Translation-Specialist LLM
• Small Language Models: Survey, Measurements, and Insights
• EuroLLM: Multilingual Language Models for Europe
• Towards Probing Speech-Specific Risks in Large Multimodal Models: A Taxonomy, Benchmark, and Insights
• Twists, Humps, and Pebbles: Multilingual Speech Recognition Models Exhibit Gender Performance Gaps
• Tower: An Open Multilingual Large Language Model for Translation-Related Tasks
• Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?
• Speech Translation with Large Language Models: An Industrial Practice
• Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models
• Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation tasks
• VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation
• NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers
• Robust Speech Recognition via Large-Scale Weak Supervision
• CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task
• FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech
• The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usage
• VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Common Voice: A Massively-Multilingual Speech Corpus
• Spoken SQuAD: A Study of Mitigating the Impact of Speech Recognition Errors on Listening Comprehension

Related

• KIT's Offline Speech Translation and Instruction Following Submission for IWSLT 2025
• NAVER LABS Europe Submission to the Instruction-following Track
• SLM: Bridge the thin gap between speech and text foundation models
• DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Data
• Virtuoso: Massive Multilingual Speech-Text Joint Semi-Supervised Learning for Text-To-Speech