Learning to Branch for Multi-Task Learning

2020/06/02 by Pengsheng Guo, Guo, Pengsheng, Chen‐Yu Lee +3 · 8 citations
Computer Science · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.2006.01895

openalex publication_date 2020/06/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Training multiple tasks jointly in one deep network yields reduced latency during inference and better performance over the single-task counterpart by sharing certain layers of a network. However, over-sharing a network could erroneously enforce over-generalization, causing negative knowledge transfer across tasks. Prior works rely on human intuition or pre-computed task relatedness scores for ad hoc branching structures. They provide sub-optimal end results and often require huge efforts for the trial-and-error process. In this work, we present an automated multi-task learning algorithm that learns where to share or branch within a network, designing an effective network topology that is directly optimized for multiple objectives across tasks. Specifically, we propose a novel tree-structured design space that casts a tree branching operation as a gumbel-softmax sampling procedure. This enables differentiable network splitting that is end-to-end trainable. We validate the proposed method on controlled synthetic data, CelebA, and Taskonomy.

Citations

• ImageNet classification with deep convolutional neural networks
• Categorical Reparameterization with Gumbel-Softmax
• An Overview of Multi-Task Learning in Deep Neural Networks
• Efficient Neural Architecture Search via Parameter Sharing
• DARTS: Differentiable Architecture Search
• Multi-Task Learning as Multi-Objective Optimization
• Which Tasks Should Be Learned Together in Multi-task Learning?
• Meta Architecture Search
• AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning

Cited by

• Simplifying Multi-Task Architectures Through Task-Specific Normalization
• A Conflict-Aware Resource Management Framework for the Computing Continuum
• Efficiently Learning Branching Networks for Multitask Algorithmic Reasoning
• 3D-Aware Multi-Task Learning with Cross-View Correlations for Dense Scene Understanding
• Neuroplasticity-inspired dynamic ANNs for multi-task demand forecasting
• Monotonic Transformation Invariant Multi-task Learning
• Revisit the Imbalance Optimization in Multi-task Learning: An Experimental Analysis
• Parameter-Efficient Multi-Task Learning via Progressive Task-Specific Adaptation

Related

• Branched Multi-Task Networks: Deciding What Layers To Share
• Tree DNN: A Deep Container Network
• Connectivity Learning in Multi-Branch Networks
• Cross-stitch Networks for Multi-task Learning
• Flexible Multi-task Networks by Learning Parameter Allocation