Objective Soups: Multilingual Multi-Task Modeling for Speech Processing

2025/08/12 by Saif, A F M, Chen, Lisha, Cui, Xiaodong +3
#Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Optimization and Control (math.OC) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2508.09228

Abstract

Training a single model for multilingual, multi-task speech processing (MSP) is severely hampered by conflicting objectives between tasks like speech recognition and translation. While multi-objective optimization (MOO) aims to align gradient updates, its effectiveness diminishes as the number of tasks grows, making it difficult to find a common descent direction. This raises a fundamental question: should highly conflicting objectives be optimized jointly or separated into a hierarchical structure? To address this question, this paper investigates three multi-objective MSP formulations, which we refer to as objective soup recipes. These formulations apply multi-objective optimization at different optimization levels to mitigate potential conflicts among all objectives. To ensure efficiency, we introduce a lightweight layer-selection mechanism that computes the conflict-avoiding gradient using only the most problematic layers, minimizing computational and memory overhead. Extensive experiments on CoVoST v2, LibriSpeech, and AISHELL-1 reveal that a bi-level recipe separating recognition and translation tasks consistently outperforms standard flat optimization. Our work demonstrates that hierarchical MOO is a more effective and scalable approach for building state-of-the-art MSP models. Our code has been released at https://github.com/afmsaif/Objective_Soups.

Citations

• Efficient First-Order Optimization on the Pareto Set for Multi-Objective Learning under Preference Guidance
• Bilevel Joint Unsupervised and Supervised Training for Automatic Speech Recognition
• FERERO: A Flexible Framework for Preference-Guided Multi-Objective Learning
• Joint Unsupervised and Supervised Training for Automatic Speech Recognition via Bilevel Optimization
• Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards
• Recon: Reducing Conflicting Gradients from the Root for Multi-Task Learning
• On Penalty-based Bilevel Gradient Descent Method
• Robust Speech Recognition via Large-Scale Weak Supervision
• Mitigating Gradient Bias in Multi-objective Learning: A Provably Convergent Stochastic Approach
• Do Current Multi-Task Optimization Methods in Deep Learning Even Help?
• mSLAM: Massively multilingual joint pre-training for speech and text
• Self-supervised Learning with Random-projection Quantizer for Speech Recognition
• In Defense of the Unitary Scalarization for Deep Multi-Task Learning
• XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale
• Joint Unsupervised and Supervised Training for Multilingual ASR
• Conflict-Averse Gradient Descent for Multi-task Learning
• Scalable and Efficient MoE Training for Multitask Multilingual Models
• Intermediate Loss Regularization for CTC-based Speech Recognition
• Joint Masked CPC and CTC Training for ASR
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• Worse WER, but Better BLEU? Leveraging Word Embedding as Intermediate in Multitask End-to-End Speech Translation
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Conformer: Convolution-augmented Transformer for Speech Recognition
• vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
• The stochastic multi-gradient algorithm for multi-objective optimization and its application to supervised machine learning
• Attention-Passing Models for Robust and Data-Efficient End-to-End Speech\n Translation
• wav2vec: Unsupervised Pre-training for Speech Recognition
• SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing
• Representation Learning with Contrastive Predictive Coding
• Multilingual End-to-End Speech Recognition with A Single Transformer on Low-Resource Languages
• Tied Multitask Learning for Neural Speech Translation
• Multilingual Speech Recognition With A Single End-To-End Model
• AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline
• Speech Recognition with Deep Recurrent Neural Networks
• Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups
• Polyglot Prompt: Multilingual Multitask PrompTraining

Related

• PolySpeech: Exploring Unified Multitask Speech Models for Competitiveness with Single-task Models
• Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
• HarmoniFuse: A Component-Selective and Prompt-Adaptive Framework for Multi-Task Speech Language Modeling
• EMMeTT: Efficient Multimodal Machine Translation Training
• Multilingual Speech Recognition using Knowledge Transfer across Learning Processes