Rare Word Recognition and Translation Without Fine-Tuning via Task Vector in Speech Models

2025/12/26 by Ruihao Jing, Jing, Ruihao, Cheng Gong +13
Computer Science · #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Natural Language Processing Techniques #Sound (cs.SD) #Speech Recognition and Synthesis #Topic Modeling #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2512.21894

openalex publication_date 2025/12/26 · openalex created_date 2025/12/30 · openalex updated_date 2026/07/28

Abstract

Rare words remain a critical bottleneck for speech-to-text systems. While direct fine-tuning improves recognition of target words, it often incurs high cost, catastrophic forgetting, and limited scalability. To address these challenges, we propose a training-free paradigm based on task vectors for rare word recognition and translation. By defining task vectors as parameter differences and introducing word-level task vector arithmetic, our approach enables flexible composition of rare-word capabilities, greatly enhancing scalability and reusability. Extensive experiments across multiple domains show that the proposed method matches or surpasses fine-tuned models on target words, improves general performance by about 5 BLEU, and mitigates catastrophic forgetting.

Citations

• Locate-and-Focus: Enhancing Terminology Translation in Speech Language Models
• Distilling a speech and music encoder with task arithmetic
• Qwen3 Technical Report
• Speech-FT: Merging Pre-trained And Fine-Tuned Speech Representation Models For Cross-Task Generalization
• CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models
• GPT-4o System Card
• Optimizing Rare Word Accuracy in Direct Speech Translation with a Retrieval-and-Demonstration Approach
• The Llama 3 Herd of Models
• FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs
• Task Arithmetic can Mitigate Synthetic-to-Real Gap in Automatic Speech Recognition
• Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch
• TIES-Merging: Resolving Interference When Merging Models
• Editing Models with Task Arithmetic
• Robust Speech Recognition via Large-Scale Weak Supervision
• Named Entity Detection and Injection for Direct Speech Translation

Related

• Optimizing Rare Word Accuracy in Direct Speech Translation with a Retrieval-and-Demonstration Approach
• Improving Rare Word Translation With Dictionaries and Attention Masking
• Multi-task Language Modeling for Improving Speech Recognition of Rare Words
• ed-cec: improving rare word recognition using asr postprocessing based on error detection and context-aware error correction
• Understanding Zero-shot Rare Word Recognition Improvements Through LLM Integration