End-to-End Multi-Task Learning with Attention

2018/03/28 by Shikun Liu, Liu, Shikun, Edward Johns +3 · 52 citations
Computer Science · Neuroscience · #Advanced Neural Network Applications #Brain Tumor Detection and Classification #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences

paper · pdf · doi:10.48550/arxiv.1803.10704

openalex publication_date 2018/03/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose a novel multi-task learning architecture, which allows learning of task-specific feature-level attention. Our design, the Multi-Task Attention Network (MTAN), consists of a single shared network containing a global feature pool, together with a soft-attention module for each task. These modules allow for learning of task-specific features from the global features, whilst simultaneously allowing for features to be shared across different tasks. The architecture can be trained end-to-end and can be built upon any feed-forward neural network, is simple to implement, and is parameter efficient. We evaluate our approach on a variety of datasets, across both image-to-image predictions and image classification tasks. We show that our architecture is state-of-the-art in multi-task learning compared to existing methods, and is also less sensitive to various weighting schemes in the multi-task loss function. Code is available at https://github.com/lorenmt/mtan.

Citations

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• A Survey on Transfer Learning
• Wide Residual Networks
• Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry\n and Semantics
• Piggyback: Adapting a Single Network to Multiple Tasks by Learning to\n Mask Weights

Cited by

• Simplifying Multi-Task Architectures Through Task-Specific Normalization
• MAGIC: Achieving Superior Model Merging via Magnitude Calibration
• An Efficient and Effective Encoder Model for Vision and Language Tasks in the Remote Sensing Domain
• Multi-Trajectory Physics-Informed Neural Networks for HJB Equations with Hard-Zero Terminal Inventory: Optimal Execution on Synthetic & SPY Data
• Task-Specific Sparse Feature Masks for Molecular Toxicity Prediction with Chemical Language Models
• NexusFlow: Unifying Disparate Tasks under Partial Supervision via Invertible Flow Networks
• 3D-Aware Multi-Task Learning with Cross-View Correlations for Dense Scene Understanding
• UniSER: A Foundation Model for Unified Soft Effects Removal
• WiCo-MG: Wireless Channel Foundation Model for Multipath Generation via Synesthesia of Machines
• DySTAN: Joint Modeling of Sedentary Activity and Social Context from Smartphone Sensors
• Expert-Guided Prompting and Retrieval-Augmented Generation for Emergency Medical Service Question Answering
• Uncertainty Modeling for Multi-Objective RTA Interception with Distillation Acceleration
• Learning-Based Blockage-Resilient Beam Training in Near-Field Terahertz Communications
• Beyond Reasoning Gains: Mitigating General Capabilities Forgetting in Large Reasoning Models
• Imitation Learning Policy based on Multi-Step Consistent Integration Shortcut Model
• NTKMTL: Mitigating Task Imbalance in Multi-Task Learning from Neural Tangent Kernel Perspective
• Heterogeneous Adversarial Play in Interactive Environments
• M2H: Multi-Task Learning with Efficient Window-Based Cross-Task Attention for Monocular Spatial Perception
• You only need 4 extra tokens: Synergistic Test-time Adaptation for LLMs
• VAO: Validation-Aligned Optimization for Cross-Task Generative Auto-Bidding
• AutoBalance: An Automatic Balancing Framework for Training Physics-Informed Neural Networks
• TokenChain: A Discrete Speech Chain via Semantic Token Modeling
• Adaptive Shared Experts with LoRA-Based Mixture of Experts for Multi-Task Learning
• Monotonic Transformation Invariant Multi-task Learning
• Revisit the Imbalance Optimization in Multi-task Learning: An Experimental Analysis
• Efficient Domain-Adaptive Multi-Task Dense Prediction with Vision Foundation Models
• GraphIFE: Rethinking Graph Imbalance Node Classification via Invariant Learning
• SSVIF: Self-Supervised Segmentation-Oriented Visible and Infrared Image Fusion
• HuLA: Prosody-Aware Anti-Spoofing with Multi-Task Learning for Expressive and Emotional Synthetic Speech
• Deep Hierarchical Learning with Nested Subspace Networks for Large Language Models
• Graph Coloring for Multi-Task Learning
• Towards Anytime Retrieval: A Benchmark for Anytime Person Re-Identification
• Temporal Representation Learning of Phenotype Trajectories for pCR Prediction in Breast Cancer
• MultiEdit: Advancing Instruction-based Image Editing on Diverse and Challenging Tasks
• MAFS: Masked Autoencoder for Infrared-Visible Image Fusion and Semantic Segmentation
• Simple Optimizers for Convex Aligned Multi-Objective Optimization
• Improving Vessel Segmentation with Multi-Task Learning and Auxiliary Data Available Only During Model Training
• Predicting Movie Success with Multi-Task Learning: A Hybrid Framework Combining GPT-Based Sentiment Analysis and SIR Propagation
• MUSE-FM: Multi-task Environment-aware Foundation Model for Wireless Communications
• Model Unmerging: Making Your Models Unmergeable for Secure Model Sharing
• Hidden Tail: Adversarial Image Causing Stealthy Resource Consumption in Vision-Language Models
• AutoScale: Linear Scalarization Guided by Multi-Task Optimization Metrics
• Can Multitask Learning Enhance Model Explainability?
• Prediction of Survival Outcomes under Clinical Presence Shift: A Joint Neural Network Architecture
• TurboTrain: Towards Efficient and Balanced Multi-Task Learning for Multi-Agent Perception and Prediction
• Seeing More with Less: Video Capsule Endoscopy with Multi-Task Learning
• Learned Off-aperture Encoding for Wide Field-of-view RGBD Imaging
• A Survey on Deep Multi-Task Learning in Connected Autonomous Vehicles
• Rep-MTL: Unleashing the Power of Representation-level Task Saliency for Multi-Task Learning
• Multi-output Deep-Supervised Classifier Chains for Plant Pathology
• DEFNet: Multitasks-based Deep Evidential Fusion Network for Blind Image Quality Assessment
• A Novel Coded Computing Approach for Distributed Multi-Task Learning

Related

• Sequential Cross Attention Based Multi-task Learning
• MILA: Multi-Task Learning from Videos via Efficient Inter-Frame Attention
• Medusa: Universal Feature Learning via Attentional Multitasking
• Cross-Task Attention Network: Improving Multi-Task Learning for Medical Imaging Applications
• Attentive Feature Reuse for Multi Task Meta learning