Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout

2020/10/14 by Chen Zhao, Jiquan Ngiam, Chen, Zhao +11 · 21 citations
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2010.06808

openalex publication_date 2020/10/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The vast majority of deep models use multiple gradient signals, typically corresponding to a sum of multiple loss terms, to update a shared set of trainable weights. However, these multiple updates can impede optimal training by pulling the model in conflicting directions. We present Gradient Sign Dropout (GradDrop), a probabilistic masking procedure which samples gradients at an activation layer based on their level of consistency. GradDrop is implemented as a simple deep layer that can be used in any deep net and synergizes with other gradient balancing approaches. We show that GradDrop outperforms the state-of-the-art multiloss methods within traditional multitask and transfer learning settings, and we discuss how GradDrop reveals links between optimal multiloss training and gradient stochasticity.

Citations

• ADADELTA: An Adaptive Learning Rate Method
• The Loss Surfaces of Multilayer Networks
• Sequence to Sequence Learning with Neural Networks
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp\n Minima
• An overview of gradient descent optimization algorithms
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal\n Networks
• Pseudo-task Augmentation: From Deep Multitask Learning to Intratask Sharing---and Back
• Multi-Task Deep Neural Networks for Natural Language Understanding
• Gradient Surgery for Multi-Task Learning
• Scalability in Perception for Autonomous Driving: Waymo Open Dataset
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Multi-Task GRPO: Reliable LLM Reasoning Across Tasks
• Simplifying Multi-Task Architectures Through Task-Specific Normalization
• MAGIC: Achieving Superior Model Merging via Magnitude Calibration
• No One Left Behind: How to Exploit the Incomplete and Skewed Multi-Label Data for Conversion Rate Prediction
• Learning to Generate Cross-Task Unexploitable Examples
• 3D-Aware Multi-Task Learning with Cross-View Correlations for Dense Scene Understanding
• Soft Conflict-Resolution Decision Transformer for Offline Multi-Task Reinforcement Learning
• Efficient Utility-Preserving Machine Unlearning with Implicit Gradient Surgery
• Beyond Reasoning Gains: Mitigating General Capabilities Forgetting in Large Reasoning Models
• Imbalanced Gradients in RL Post-Training of Multi-Task LLMs
• NTKMTL: Mitigating Task Imbalance in Multi-Task Learning from Neural Tangent Kernel Perspective
• VAO: Validation-Aligned Optimization for Cross-Task Generative Auto-Bidding
• Monotonic Transformation Invariant Multi-task Learning
• Revisit the Imbalance Optimization in Multi-task Learning: An Experimental Analysis
• MAESTRO: Task-Relevant Optimization via Adaptive Feature Enhancement and Suppression for Multi-task 3D Perception
• GCond: Gradient Conflict Resolution via Accumulation-based Stabilization for Large-Scale Multi-Task Learning
• Non-conflicting Energy Minimization in Reinforcement Learning based Robot Control
• AutoScale: Linear Scalarization Guided by Multi-Task Optimization Metrics
• TriForecaster: A Mixture of Experts Framework for Multi-Region Electric Load Forecasting with Tri-dimensional Specialization
• Rep-MTL: Unleashing the Power of Representation-level Task Saliency for Multi-Task Learning
• Game-Theoretic Gradient Control for Robust Neural Network Training

Related

• Dropout and Random Gradient Masking Are Asymptotically Equivalent in Large ResNets
• Dropout against Deep Leakage from Gradients
• Beyond Losses Reweighting: Empowering Multi-Task Learning via the Generalization Perspective
• Gradient-Sign Masking for Task Vector Transport Across Pre-Trained Models
• Gradient Sparsification For Masked Fine-Tuning of Transformers