Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry\n and Semantics
2017/05/19 by Alex Kendall, Yarin Gal, Kendall, Alex +3 · 182 citations
Computer Science · #Domain Adaptation and Few-Shot Learning #Advanced Neural Network Applications #Human Pose and Action Recognition
paper · pdf · doi:10.48550/arxiv.1705.07115
Abstract
Numerous deep learning applications benefit from multi-task learning with\nmultiple regression and classification objectives. In this paper we make the\nobservation that the performance of such systems is strongly dependent on the\nrelative weighting between each task's loss. Tuning these weights by hand is a\ndifficult and expensive process, making multi-task learning prohibitive in\npractice. We propose a principled approach to multi-task deep learning which\nweighs multiple loss functions by considering the homoscedastic uncertainty of\neach task. This allows us to simultaneously learn various quantities with\ndifferent units or scales in both classification and regression settings. We\ndemonstrate our model learning per-pixel depth regression, semantic and\ninstance segmentation from a monocular input image. Perhaps surprisingly, we\nshow our model can learn multi-task weightings and outperform separate models\ntrained individually on each task.\n
Cited by
- When Does Multi-Task Learning Fail? Quantifying Data Imbalance and Task Independence in Metal Alloy Property Prediction
- OmniVAE: An Audio-Video VAE with Cross-Modal Alignment for Joint Generation
- AutoFed: Personalized Federated Traffic Prediction via Adaptive Prompt
- Lantern: Conflict-Aware Gradient Blending for Physics-Guided Diffusion Models in Calorimeter Simulation
- CuraWeb: Joint Optimization of Quality, Redundancy, and Diversity for Web-Scale Pretraining Data
- Multi-Task GRPO: Reliable LLM Reasoning Across Tasks
- ShinyNeRF: Digitizing Anisotropic Appearance in Neural Radiance Fields
- Fast SAM2 with Text-Driven Token Pruning
- Model Merging via Multi-Teacher Knowledge Distillation
- Human Motion Estimation with Everyday Wearables
- Simplifying Multi-Task Architectures Through Task-Specific Normalization
- Mixture-of-Experts with Gradient Conflict-Driven Subspace Topology Pruning for Emergent Modularity
- MAGIC: Achieving Superior Model Merging via Magnitude Calibration
- Rectification Reimagined: A Unified Mamba Model for Image Correction and Rectangling with Prompts
- Text2Graph VPR: A Text-to-Graph Expert System for Explainable Place Recognition in Changing Environments
- Achieving angular-momentum conservation with physics-informed neural networks in computational relativistic spin hydrodynamics
- Metanetworks as Regulatory Operators: Learning to Edit for Requirement Compliance
- Learning Model Parameter Dynamics in a Combination Therapy for Bladder Cancer from Sparse Biological Data
- KD360-VoxelBEV: LiDAR and 360-degree Camera Cross Modality Knowledge Distillation for Bird's-Eye-View Segmentation
- Adaptive Multimodal Person Recognition: A Robust Framework for Handling Missing Modalities
- Self-adaptive physics-informed neural network for forward and inverse problems in heterogeneous porous flow
- TAT: Task-Adaptive Transformer for All-in-One Medical Image Restoration
- Physics-Informed Cross-Learning for Seismic Acoustic Impedance Inversion and Wavelet Extraction
- Optimal transport unlocks end-to-end learning for single-molecule localization
- AutoSeM: Automatic Task Selection and Mixing in Multi-Task Learning
- OmniFD: A Unified Model for Versatile Face Forgery Detection
- Fine-tuning an ECG Foundation Model to Predict Coronary CT Angiography Outcomes
- LUMOS: Large User MOdels for User Behavior Prediction
- Multimodal Robust Prompt Distillation for 3D Point Cloud Models
- 3D-Aware Multi-Task Learning with Cross-View Correlations for Dense Scene Understanding
- Segment-Wise Flow Matching for Vision-Aided mmWave V2I Beam Prediction
- CNN-Based Camera Pose Estimation and Localisation of Scan Images for Aircraft Visual Inspection
- From Simulations to Surveys: Domain Adaptation for Galaxy Observations
- PhysGS: Bayesian-Inferred Gaussian Splatting for Physical Property Estimation
- Auxiliary Gene Learning: Spatial Gene Expression Estimation by Auxiliary Gene Selection
- Collaborative Management for Chronic Diseases and Depression: A Double Heterogeneity-based Multi-Task Learning Method
- Mem-MLP: Real-Time 3D Human Motion Generation from Sparse Inputs
- DySTAN: Joint Modeling of Sedentary Activity and Social Context from Smartphone Sensors
- Degree-of-freedom and optimization-dynamic effects on the observability of Kuramoto-Sivashinsky systems
- Semi-Supervised Multi-Task Learning for Interpretable Quality As- sessment of Fundus Images
- A Decoupled Uncertainty Model for MRI Segmentation Quality Estimation
- Gradient Surgery for Multi-Task Learning
- MMSense: Adapting Vision-based Foundation Model for Multi-task Multi-modal Wireless Sensing
- DINOv3 as a Frozen Encoder for CRPS-Oriented Probabilistic Rainfall Nowcasting
- Beyond MSE: Ordinal Cross-Entropy for Probabilistic Time Series Forecasting
- Towards a Machine Learning Solution for Hubble Tension: Physics-Informed Neural Network (PINN) Analysis of Tsallis Holographic Dark Energy in Presence of Neutrinos
- Trend-Aware Multi-Task Learning for Short-Term Energy Forecasting
- A Negotiation-Based Multi-Agent Reinforcement Learning Approach for Dynamic Scheduling of Reconfigurable Manufacturing Systems
- COGNOS: Universal Enhancement for Time Series Anomaly Detection via Constrained Gaussian-Noise Optimization and Smoothing
- Which Tasks Should Be Learned Together in Multi-task Learning?
- Uncertainty Modeling for Multi-Objective RTA Interception with Distillation Acceleration
- Scheduling the Off-Diagonal Weingarten Loss of Neural SDFs for CAD Models
- Object Detection as an Optional Basis: A Graph Matching Network for Cross-View UAV Localization
- Balanced Multimodal Learning via Mutual Information
- Driven to Distraction: Self-Supervised Distractor Learning for Robust Monocular Visual Odometry in Urban Environments
- DEFT: Detection Embeddings for Tracking
- SCNet: A Generalized Attention-based Model for Crack Fault Segmentation
- When Fish Look Alike: Tracking Identities with Dual-branch Elasticity
- When Top-K Misses the Decision: Tool-Call Drift in Multi-Teacher On-Policy Distillation
- Objects are Different: Flexible Monocular 3D Object Detection
- Libra R-CNN: Towards Balanced Learning for Object Detection
- Weakly Supervised Learning of Instance Segmentation with Inter-pixel Relations
- Belief-Guided Decision Making with Uncertainty Gating in the Game of Go
- Multi-Channel Attention Selection GANs for Guided Image-to-Image Translation
- Gated-SCNN: Gated Shape CNNs for Semantic Segmentation
- RGB-based Semantic Segmentation Using Self-Supervised Depth Pre-Training
- Uncertain Knowledge Graph Completion via Semi-Supervised Confidence Distribution Learning
- Multimodal Motion Prediction with Stacked Transformers
- Decoupled MeanFlow: Turning Flow Models into Flow Maps for Accelerated Sampling
- IAEmu: Learning Galaxy Intrinsic Alignment Correlations
- KongNet: A Multi-headed Deep Learning Model for Detection and Classification of Nuclei in Histopathology Images
- LoMix: Learnable Weighted Multi-Scale Logits Mixing for Medical Image Segmentation
- Generating Auxiliary Tasks with Reinforcement Learning
- Estimation of Fireproof Structure Class and Construction Year for Disaster Risk Assessment
- EEGReXferNet: A Lightweight Gen-AI Framework for EEG Subspace Reconstruction via Cross-Subject Transfer Learning and Channel-Aware Embedding
- Efficient Utility-Preserving Machine Unlearning with Implicit Gradient Surgery
- Beyond Reasoning Gains: Mitigating General Capabilities Forgetting in Large Reasoning Models
- LightsOut: Diffusion-based Outpainting for Enhanced Lens Flare Removal
- Evaluating Bayesian Deep Learning Methods for Semantic Segmentation
- LLM-Integrated Bayesian State Space Models for Multimodal Time-Series Forecasting
- Multi-modal Co-learning for Earth Observation: Enhancing single-modality models via modality collaboration
- Seabed-Net: A multi-task network for joint bathymetry estimation and seabed classification from remote sensing imagery in shallow waters
- NTKMTL: Mitigating Task Imbalance in Multi-Task Learning from Neural Tangent Kernel Perspective
- M2H: Multi-Task Learning with Efficient Window-Based Cross-Task Attention for Monocular Spatial Perception
- U-LAG: Uncertainty-Aware, Lag-Adaptive Goal Retargeting for Robotic Manipulation
- Auxiliary Tasks Speed Up Learning PointGoal Navigation
- Unconditional Human Motion and Shape Generation via Balanced Score-Based Diffusion
- Visual Odometry with Transformers
- Wrapped Loss Function for Regularizing Nonconforming Residual\n Distributions
- CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance-Diversity Optimization
- Scalable Offline Metrics for Autonomous Driving
- STEPER: Step-wise Knowledge Distillation for Enhancing Reasoning Ability in Multi-Step Retrieval-Augmented Language Models
- VAO: Validation-Aligned Optimization for Cross-Task Generative Auto-Bidding
- SONA: Learning Conditional, Unconditional, and Mismatching-Aware Discriminator
- Learning More with Less: A Generalizable, Self-Supervised Framework for Privacy-Preserving Capacity Estimation with EV Charging Data
- Learning more physically realistic dynamics in machine-learning based weather forecasting with latent-space constraints
- AortaDiff: A Unified Multitask Diffusion Framework For Contrast-Free AAA Imaging
- Graph2Region: Efficient Graph Similarity Learning with Structure and Scale Restoration
- Articulation-Informed ASR: Integrating Articulatory Features into ASR via Auxiliary Speech Inversion and Cross-Attention Fusion
- Adapting SAM with Dynamic Similarity Graphs for Few-Shot Parameter-Efficient Small Dense Object Detection: A Case Study of Chickpea Pods in Field Conditions
- Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAs
- VSSFlow: Unifying Video-conditioned Sound and Speech Generation via Joint Learning
- DyMoDreamer: World Modeling with Dynamic Modulation
- Monotonic Transformation Invariant Multi-task Learning
- Revisit the Imbalance Optimization in Multi-task Learning: An Experimental Analysis
- Efficient Domain-Adaptive Multi-Task Dense Prediction with Vision Foundation Models
- AW-EL-PINNs: A Multi-Task Learning Physics-Informed Neural Network for Euler-Lagrange Systems in Optimal Control Problems
- Uncertainty-aware Joint Salient Object and Camouflaged Object Detection
- Metric-based Regularization and Temporal Ensemble for Multi-task\n Learning using Heterogeneous Unsupervised Tasks
- When Does Self-supervision Improve Few-shot Learning?
- Context-aware sequence-to-function model of human gene regulation
- SSVIF: Self-Supervised Segmentation-Oriented Visible and Infrared Image Fusion
- Wav2Arrest 2.0: Long-Horizon Cardiac Arrest Prediction with Time-to-Event Modeling, Identity-Invariance, and Pseudo-Lab Alignment
- Process-Informed Forecasting of Complex Thermal Dynamics in Pharmaceutical Manufacturing
- Shared Neural Space: Unified Precomputed Feature Encoding for Multi-Task and Cross Domain Vision
- Parameter-Efficient Multi-Task Learning via Progressive Task-Specific Adaptation
- Harnessing the Potential of Optimizing Data Mixtures via Bayesian Domain Reweighting
- A transformer-based multi-task deep learning model for urban livability evaluation by fusing remote sensing and textual geospatial data
- Hybrid Physics-Informed and Bayesian Modeling of Single-Nanoparticle–Cell Adhesion Kinetics under Cytoskeletal Perturbation
- Ada-TransGNN: An Air Quality Prediction Model Based On Adaptive Graph Convolutional Networks
- Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation
- Uncertainty-Aware Physically-Guided Proxy Tasks for Unseen Domain Face Anti-spoofing
- Uncertainty-Aware Deep Calibrated Salient Object Detection
- SGAligner++: Cross-Modal Language-Aided 3D Scene Graph Alignment
- LCMF: Lightweight Cross-Modality Mambaformer for Embodied Robotics VQA
- Live-E2T: Real-time Threat Monitoring in Video via Deduplicated Event Reasoning and Chain-of-Thought
- Multimodal Radio and Vision Fusion for Robust Localization in Urban V2I Communications
- MAESTRO: Task-Relevant Optimization via Adaptive Feature Enhancement and Suppression for Multi-task 3D Perception
- Deep Hierarchical Learning with Nested Subspace Networks for Large Language Models
- Graph Coloring for Multi-Task Learning
- On the Simplification of Neural Network Architectures for Predictive Process Monitoring
- AHA -- Predicting What Matters Next: Online Highlight Detection Without Looking Ahead
- Towards Anytime Retrieval: A Benchmark for Anytime Person Re-Identification
- MultiEdit: Advancing Instruction-based Image Editing on Diverse and Challenging Tasks
- Multi-Sensor 3D Object Box Refinement for Autonomous Driving
- Semi-MoE: Mixture-of-Experts meets Semi-Supervised Histopathology Segmentation
- Temporal Attentive Alignment for Large-Scale Video Domain Adaptation
- MAFS: Masked Autoencoder for Infrared-Visible Image Fusion and Semantic Segmentation
- Topic-Guided Reinforcement Learning with LLMs for Enhancing Multi-Document Summarization
- LGE-Guided Cross-Modality Contrastive Learning for Gadolinium-Free Cardiomyopathy Screening in Cine CMR
- Renovating Parsing R-CNN for Accurate Multiple Human Parsing
- Joint Learning using Mixture-of-Expert-Based Representation for Enhanced Speech Generation and Robust Emotion Recognition
- GCond: Gradient Conflict Resolution via Accumulation-based Stabilization for Large-Scale Multi-Task Learning
- Robust and Adaptive Spectral Method for Representation Multi-Task Learning with Contamination
- Learning to Route: Per-Sample Adaptive Routing for Multimodal Multitask Prediction
- SuperSNN: A Hardware-Aware Framework for Physically Realizable, High-Performance Superconducting Spiking Neural Network Chips
- Exploiting Unlabeled Structures through Task Consistency Training for Versatile Medical Image Segmentation
- Quantum-Enhanced Multi-Task Learning with Learnable Weighting for Pharmacokinetic and Toxicity Prediction
- Improving Vessel Segmentation with Multi-Task Learning and Auxiliary Data Available Only During Model Training
- A Neural Network Approach to Multi-radionuclide TDCR Beta Spectroscopy
- Predicting Movie Success with Multi-Task Learning: A Hybrid Framework Combining GPT-Based Sentiment Analysis and SIR Propagation
- Multimodal Generative Flows for LHC Jets
- MVTrajecter: Multi-View Pedestrian Tracking with Trajectory Motion Cost and Trajectory Appearance Cost
- To New Beginnings: A Survey of Unified Perception in Autonomous Vehicle Software
- MPFormer: Adaptive Framework for Industrial Multi-Task Personalized Sequential Retriever
- Enhancing Mamba Decoder with Bidirectional Interaction in Multi-Task Dense Prediction
- Learning Probabilistic Ordinal Embeddings for Uncertainty-Aware Regression
- Distance-informed Neural Processes
- Joint Spatial-Temporal Optimization for Stereo 3D Object Tracking
- AutoScale: Linear Scalarization Guided by Multi-Task Optimization Metrics
- SUB-Depth: Self-distillation and Uncertainty Boosting Self-supervised Monocular Depth Estimation
- FAMNet: Integrating 2D and 3D Features for Micro-expression Recognition via Multi-task Learning and Hierarchical Attention
- Bounding Box Regression with Uncertainty for Accurate Object Detection
- NeMo: A Neuron-Level Modularizing-While-Training Approach for Decomposing DNN Models
- When Experts Disagree: Characterizing Annotator Variability for Vessel Segmentation in DSA Images
- Self-Supervised Temporal Super-Resolution of Energy Data using Generative Adversarial Transformer
- Visually Aware Skip-Gram for Image Based Recommendations
- AANet: Attribute Attention Network for Person Re-Identifications
- Multi-dimensional Neural Decoding with Orthogonal Representations for Brain-Computer Interfaces
- Instance-Level Task Parameters: A Robust Multi-task Weighting Framework
- Can Multitask Learning Enhance Model Explainability?
- FDC-Net: Rethinking the association between EEG artifact removal and multi-dimensional affective computing
- Conditional Latent Diffusion Models for Zero-Shot Instance Segmentation
- Ultralight Polarity-Split Neuromorphic SNN for Event-Stream Super-Resolution
- Learning Dynamics of Meta-Learning in Small Model Pretraining
- One-Vote Veto: Semi-Supervised Learning for Low-Shot Glaucoma Diagnosis
- Seeing More with Less: Video Capsule Endoscopy with Multi-Task Learning
- Hierarchical Attentive Recurrent Tracking
- A Survey on Deep Multi-Task Learning in Connected Autonomous Vehicles
- ArtSeek: Deep artwork understanding via multimodal in-context reasoning and late interaction retrieval
- Deep Likelihood Network for Image Restoration with Multiple Degradation Levels
- Rep-MTL: Unleashing the Power of Representation-level Task Saliency for Multi-Task Learning
Related