TICL: Text-Embedding KNN For Speech In-Context Learning Unlocks Speech Recognition Abilities of Large Multimodal Models

2025/09/16 by Zheng, Haolong, Yegorova, Yekaterina, Hasegawa-Johnson, Mark · 1 citation
#Artificial Intelligence (cs.AI) #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Multimedia (cs.MM) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2509.13395

Abstract

Speech foundation models have recently demonstrated the ability to perform Speech In-Context Learning (SICL). Selecting effective in-context examples is crucial for SICL performance, yet selection methodologies remain underexplored. In this work, we propose Text-Embedding KNN for SICL (TICL), a simple pipeline that uses semantic context to enhance off-the-shelf large multimodal models' speech recognition ability without fine-tuning. Across challenging automatic speech recognition tasks, including accented English, multilingual speech, and children's speech, our method enables models to surpass zero-shot performance with up to 84.7% relative WER reduction. We conduct ablation studies to show the robustness and efficiency of our method.

Citations

• Omnilingual ASR: Open-Source Multilingual Speech Recognition for 1600+ Languages
• In-Context Learning Boosts Speech Recognition via Human-like Adaptation to Speakers and Language Varieties
• Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs
• OWLS: Scaling Laws for Multilingual Speech Recognition and Translation Models
• M2R-Whisper: Multi-stage and Multi-scale Retrieval Augmentation for Enhancing Whisper
• Qwen2-Audio Technical Report
• FASA: a Flexible and Automatic Speech Aligner for Extracting High-quality Aligned Children Speech Data
• GLOBE: A High-quality English Corpus with Global Accents for Zero-shot Speaker Adaptive Text-to-Speech
• Benchmarking Children's ASR with Supervised and Self-supervised Speech Foundation Models
• Many-Shot In-Context Learning
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
• My Science Tutor (MyST) -- A Large Corpus of Children's Conversational Speech
• Can Whisper perform speech-based in-context learning?
• In-context Learning as Maintaining Coherency: A Study of On-the-fly Machine Translation Using Large Language Models
• A Survey on In-context Learning
• Robust Speech Recognition via Large-Scale Weak Supervision
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
• Calibrate Before Use: Improving Few-Shot Performance of Language Models
• Language Models are Few-Shot Learners
• Common Voice: A Massively-Multilingual Speech Corpus

Cited by

• TICL+: A Case Study On Speech In-Context Learning for Children's Speech Recognition

Related

• TICL+: A Case Study On Speech In-Context Learning for Children's Speech Recognition
• In-Context Learning Boosts Speech Recognition via Human-like Adaptation to Speakers and Language Varieties
• Exploring In-Context Learning of Textless Speech Language Model for Speech Classification Tasks
• End-to-End Speech Recognition Contextualization with Large Language Models
• SMILE: Speech Meta In-Context Learning for Low-Resource Language Automatic Speech Recognition