Uniform Loss vs. Specialized Optimization: A Comparative Analysis in Multi-Task Learning

2025/05/15 by Gabriel S. Gama, Gama, Gabriel S., Valdir Grassi +1
Computer Science · #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2505.10347

openalex publication_date 2025/05/15 · openalex created_date 2025/10/15 · openalex updated_date 2026/08/01

Abstract

Specialized Multi-Task Optimizers (SMTOs) balance task learning in Multi-Task Learning by addressing issues like conflicting gradients and differing gradient norms, which hinder equal-weighted task training. However, recent critiques suggest that equally weighted tasks can achieve competitive results compared to SMTOs, arguing that previous SMTO results were influenced by poor hyperparameter optimization and lack of regularization. In this work, we evaluate these claims through an extensive empirical evaluation of SMTOs, including some of the latest methods, on more complex multi-task problems to clarify this behavior. Our findings indicate that SMTOs perform well compared to uniform loss and that fixed weights can achieve competitive performance compared to SMTOs. Furthermore, we demonstrate why uniform loss perform similarly to SMTOs in some instances. The source code is available at https://github.com/Gabriel-SGama/UnitScal_vs_SMTOs.

Citations

• GO4Align: Group Optimization for Multi-Task Alignment
• Bayesian Uncertainty for Gradient Aggregation in Multi-Task Learning
• Scalarization for Multi-Task and Multi-Domain Learning at Scale
• Revisiting Scalarization in Multi-Task Learning: A Theoretical Perspective
• Dual-Balancing for Multi-Task Learning
• FAMO: Fast Adaptive Multitask Optimization
• Causes and Cures for Interference in Multilingual Translation
• Do Current Multi-Task Optimization Methods in Deep Learning Even Help?
• Leveraging convergence behavior to balance conflicting tasks in multi-task learning
• Auto-Lambda: Disentangling Dynamic Task Relationships
• Multi-Task Learning as a Bargaining Game
• In Defense of the Unitary Scalarization for Deep Multi-Task Learning
• Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learning
• Conflict-Averse Gradient Descent for Multi-task Learning
• Efficiently Identifying Task Groupings for Multi-Task Learning
• RotoGrad: Gradient Homogenization in Multitask Learning
• Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout
• Follow the bisector: a simple method for multi-objective optimization
• Understanding and Improving Information Transfer in Multi-Task Learning
• Meta-Learning in Neural Networks: A Survey
• Gradient Surgery for Multi-Task Learning
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Which Tasks Should Be Learned Together in Multi-task Learning?
• Fast Graph Representation Learning with PyTorch Geometric
• Multi-Task Learning as Multi-Objective Optimization
• End-to-End Multi-Task Learning with Attention
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• An Alternative View: When Does SGD Escape Local Minima?
• Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
• Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry\n and Semantics
• Neural Message Passing for Quantum Chemistry
• MoleculeNet: A Benchmark for Molecular Machine Learning
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp\n Minima
• Cross-stitch Networks for Multi-task Learning
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• Deep Residual Learning for Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• A Model of Inductive Bias Learning
• Gradient-based learning applied to document recognition

Related

• SLAW: Scaled Loss Approximate Weighting for Efficient Multi-Task Learning
• Beyond Losses Reweighting: Empowering Multi-Task Learning via the Generalization Perspective
• Revisit the Imbalance Optimization in Multi-task Learning: An Experimental Analysis
• Improved optimization strategies for deep Multi-Task Networks
• Can Optimization Trajectories Explain Multi-Task Transfer?