Continual Speech Learning with Fused Speech Features

2025/06/02 by Wang, Guitao, Zhao, Jinming, Yang, Hao +3
#Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2506.01496

Abstract

Rapid growth in speech data demands adaptive models, as traditional static methods fail to keep pace with dynamic and diverse speech information. We introduce continuous speech learning, a new set-up targeting at bridging the adaptation gap in current speech models. We use the encoder-decoder Whisper model to standardize speech tasks into a generative format. We integrate a learnable gated-fusion layer on the top of the encoder to dynamically select task-specific features for downstream tasks. Our approach improves accuracy significantly over traditional methods in six speech processing tasks, demonstrating gains in adapting to new speech tasks without full retraining.

Citations

• Continual Learning for Large Language Models: A Survey
• Continual Learning with Pre-Trained Models: A Survey
• Towards Event Extraction from Speech with Contextual Clues
• Investigating Pre-trained Audio Encoders in the Low-Resource Condition
• A Comprehensive Survey of Continual Learning: Theory, Method and Application
• Robust Speech Recognition via Large-Scale Weak Supervision
• Continual Learning of Natural Language Processing Tasks: A Survey
• Towards Relation Extraction From Speech
• Learning to Prompt for Continual Learning
• Recent Advances of Continual Learning in Computer Vision: An Overview
• SUPERB: Speech processing Universal PERformance Benchmark
• Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining
• Dark Experience for General Continual Learning: a Strong, Simple Baseline
• Voxceleb: Large-scale speaker verification in the wild
• Speech Model Pre-training for End-to-End Spoken Language Understanding
• Experience Replay for Continual Learning
• Efficient Lifelong Learning with A-GEM
• Overcoming catastrophic forgetting with hard attention to the task
• Gradient Episodic Memory for Continual Learning
• Learning without Forgetting

Related

• Continuously Learning New Words in Automatic Speech Recognition
• UniFlow: Unifying Speech Front-End Tasks via Continuous Generative Modeling
• WhisTLE: Deeply Supervised, Text-Only Domain Adaptation for Pretrained Speech Recognition Transformers
• Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Data
• Less Forgetting for Better Generalization: Exploring Continual-learning Fine-tuning Methods for Speech Self-supervised Representations