Which Tasks Should Be Learned Together in Multi-task Learning?

2019/05/18 by Trevor Standley, Standley, Trevor, Amir Zamir +10 · 70 citations
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Domain Adaptation and Few-Shot Learning #cs.CV

paper · pdf · doi:10.48550/arxiv.1905.07553

Presented to ICML 2020 See project website at http://taskgrouping.stanford.edu/

arxiv created 2020/09/03 · arxiv updated 2020/09/04

Abstract

Many computer vision applications require solving multiple tasks in real-time. A neural network can be trained to solve multiple tasks simultaneously using multi-task learning. This can save computation at inference time as only a single network needs to be evaluated. Unfortunately, this often leads to inferior overall performance as task objectives can compete, which consequently poses the question: which tasks should and should not be learned together in one network when employing multi-task learning? We study task cooperation and competition in several different learning settings and propose a framework for assigning tasks to a few neural networks such that cooperating tasks are computed by the same neural network, while competing tasks are computed by different networks. Our framework offers a time-accuracy trade-off and can produce better accuracy using less inference time than not only a single large multi-task neural network but also many single-task networks.

Citations

• Designing Neural Network Architectures using Reinforcement Learning
• Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
• An Overview of Multi-Task Learning in Deep Neural Networks
• A Survey on Multi-Task Learning
• Regularizing Deep Multi-Task Networks using Orthogonal Gradients

Cited by

• Multi-Task Learning for Metal Alloy Property Prediction: An Empirical Study of Negative Transfer and Mitigation Strategies
• Simplifying Multi-Task Architectures Through Task-Specific Normalization
• Mixture-of-Experts with Gradient Conflict-Driven Subspace Topology Pruning for Emergent Modularity
• CrashChat: A Multimodal Large Language Model for Multitask Traffic Crash Video Analysis
• Understanding, Accelerating, and Improving MeanFlow Training
• AttnRegDeepLab: A Two-Stage Decoupled Framework for Interpretable Embryo Fragmentation Grading
• Cross-Learning from Scarce Data via Multi-Task Constrained Optimization
• Dynamic Model Selection for Trajectory Prediction via Pairwise Ranking and Meta-Features
• AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning
• Language-Conditioned Representations and Mixture-of-Experts Policy for Robust Multi-Task Robotic Manipulation
• Multi-Task Reinforcement Learning with Context-based Representations
• Generating Auxiliary Tasks with Reinforcement Learning
• NTKMTL: Mitigating Task Imbalance in Multi-Task Learning from Neural Tangent Kernel Perspective
• Heterogeneous Adversarial Play in Interactive Environments
• Quantifying Dataset Similarity to Guide Transfer Learning
• Probabilistic Hyper-Graphs using Multiple Randomly Masked Autoencoders for Semi-supervised Multi-modal Multi-task Learning
• 12-in-1: Multi-Task Vision and Language Representation Learning
• Meta-Learning Theory-Informed Inductive Biases using Deep Kernel Gaussian Processes
• Neuroplasticity-inspired dynamic ANNs for multi-task demand forecasting
• Real-Aware Residual Model Merging for Deepfake Detection
• United We Stand: Towards End-to-End Log-based Fault Diagnosis via Interactive Multi-Task Learning
• Monotonic Transformation Invariant Multi-task Learning
• Revisit the Imbalance Optimization in Multi-task Learning: An Experimental Analysis
• Depth Edge Alignment Loss: DEALing with Depth in Weakly Supervised Semantic Segmentation
• Graph Coloring for Multi-Task Learning
• Spatially distributed antecedent rainfall thresholds for landslide occurrence: a multitask machine learning modelling approach
• Efficiently Identifying Task Groupings for Multi-Task Learning
• Robust and Adaptive Spectral Method for Representation Multi-Task Learning with Contamination
• Learning to Route: Per-Sample Adaptive Routing for Multimodal Multitask Prediction
• Improving Vessel Segmentation with Multi-Task Learning and Auxiliary Data Available Only During Model Training
• DivMerge: A divergence-based model merging method for multi-tasking
• What makes instance discrimination good for transfer learning?
• AutoScale: Linear Scalarization Guided by Multi-Task Optimization Metrics
• Cross-Cancer Knowledge Transfer in WSI-based Prognosis Prediction
• Batching-Aware Joint Model Onloading and Offloading for Hierarchical Multi-Task Inference
• Can Multitask Learning Enhance Model Explainability?
• Distribution-Specific Learning for Joint Salient and Camouflaged Object Detection
• Slice or the Whole Pie? Utility Control for AI Models
• Learning Dynamics of Meta-Learning in Small Model Pretraining
• MTL-UE: Learning to Learn Nothing for Multi-Task Learning
• Rep-MTL: Unleashing the Power of Representation-level Task Saliency for Multi-Task Learning
• Learning to Branch for Multi-Task Learning
• Multi-Task Adversarial Attack
• A High Magnifications Histopathology Image Dataset for Oral Squamous Cell Carcinoma Diagnosis and Prognosis
• CDC: Causal Domain Clustering for Multi-Domain Recommendation
• XiYan-SQL: A Novel Multi-Generator Framework For Text-to-SQL
• An Audio-centric Multi-task Learning Framework for Streaming Ads Targeting on Spotify
• StableMTL: Repurposing Latent Diffusion Models for Multi-Task Learning from Partially Annotated Synthetic Datasets
• Gradient Similarity Surgery in Multi-Task Deep Learning
• Policy Search, Retrieval, and Composition via Task Similarity in Collaborative Agentic Systems
• Multi-task Learning with Active Learning for Arabic Offensive Speech Detection
• MINT: Multimodal Instruction Tuning with Multimodal Interaction Grouping
• Navigating the Accuracy-Size Trade-Off with Flexible Model Merging
• DeepChest: Dynamic Gradient-Free Task Weighting for Effective Multi-Task Learning in Chest X-ray Classification
• Measuring Fine-Grained Relatedness in Multitask Learning via Data Attribution
• Measure Domain's Gap: A Similar Domain Selection Principle for Multi-Domain Recommendation
• Multi-Task Learning with Deep Neural Networks: A Survey
• Multi-task Learning For Joint Action and Gesture Recognition
• Bridging Speech Emotion Recognition and Personality: Dataset and Temporal Interaction Condition Network
• Uniform Loss vs. Specialized Optimization: A Comparative Analysis in Multi-Task Learning
• BoundarySeg:An Embarrassingly Simple Method To Boost Medical Image Segmentation Performance for Low Data Regimes
• Measuring and Harnessing Transference in Multi-Task Learning
• CICADA: Cross-Domain Interpretable Coding for Anomaly Detection and Adaptation in Multivariate Time Series
• Parameter-efficient Quantum Multi-task Learning
• Multi-Objective Ranking for Live-Streaming: Balancing Fresh and Delayed Signals with Segment-Aware Targeting
• Radar4D-VLM: Proposal-Grounded Temporal 4D Radar Reasoning Across Frozen Language Models
• Single-Input Multi-Output Model Merging: Leveraging Foundation Models for Dense Multi-Task Learning
• Efficient Multi-Task Modeling through Automated Fusion of Trained Models
• Learning Optimal Prompt Ensemble for Multi-source Visual Prompt Transfer
• InvNeRF-Seg: Fine-Tuning a Pre-Trained NeRF for 3D Object Segmentation

Related

• Efficiently Identifying Task Groupings for Multi-Task Learning
• Multi-Task Learning for Visual Scene Understanding
• Multi-Task Learning with Deep Neural Networks: A Survey
• Navigating the Trade-Off between Multi-Task Learning and Learning to Multitask in Deep Neural Networks
• When Multi-Task Learning Meets Partial Supervision: A Computer Vision Review