Orthogonal Gradient Descent for Continual Learning

2019/10/15 by Mehrdad Farajtabar, Navid Azizan, Farajtabar, Mehrdad +5 · 35 citations
Computer Science · Medicine · #COVID-19 diagnosis using AI #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.1910.07104

openalex publication_date 2019/10/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Neural networks are achieving state of the art and sometimes super-human performance on learning tasks across a variety of domains. Whenever these problems require learning in a continual or sequential manner, however, neural networks suffer from the problem of catastrophic forgetting; they forget how to solve previous tasks after being trained on a new task, despite having the essential capacity to solve both tasks if they were trained on both simultaneously. In this paper, we propose to address this issue from a parameter space perspective and study an approach to restrict the direction of the gradient updates to avoid forgetting previously-learned data. We present the Orthogonal Gradient Descent (OGD) method, which accomplishes this goal by projecting the gradients from new tasks onto a subspace in which the neural network output on previous task does not change and the projected gradient is still in a useful direction for learning the new task. Our approach utilizes the high capacity of a neural network more efficiently and does not require storing the previously learned data that might raise privacy concerns. Experiments on common benchmarks reveal the effectiveness of the proposed OGD method.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions.
• ImageNet classification with deep convolutional neural networks
• Learning without Forgetting
• Overcoming Catastrophic Forgetting by Incremental Moment Matching
• Continuous Adaptation via Meta-Learning in Nonstationary and Competitive\n Environments
• A Convergence Theory for Deep Learning via Over-Parameterization
• Experience Replay for Continual Learning
• Towards Federated Learning at Scale: System Design
• Measuring Catastrophic Forgetting in Neural Networks
• Variational Prototype Replays for Continual Learning
• Neurogenesis deep learning: Extending deep networks to accommodate new classes

Cited by

• Merge before Forget: A Single LoRA Continual Learning via Continual Merging
• Breaking the Synthetic-Real Domain Shortcut for Training-Free Generative Replay-based Class Incremental Learning
• Learning continually with representational drift
• LibContinual: A Comprehensive Library towards Realistic Continual Learning
• FedDPC : Handling Data Heterogeneity and Partial Client Participation in Federated Learning
• Mixture-of-Experts with Gradient Conflict-Driven Subspace Topology Pruning for Emergent Modularity
• Sequencing to Mitigate Catastrophic Forgetting in Continual Learning
• Bridging Streaming Continual Learning via In-Context Large Tabular Models
• CIP-Net: Continual Interpretable Prototype-based Network
• Resolving Conflicts in Lifelong Learning via Aligning Updates in Subspaces
• Geometric-disentangelment Unlearning
• Dual-LoRA and Quality-Enhanced Pseudo Replay for Multimodal Continual Food Learning
• Neuro-Logic Lifelong Learning
• Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
• Low-Resource Dialect Adaptation of Large Language Models: A French Dialect Case-Study
• Towards Error Centric Intelligence I, Beyond Observational Learning
• OPLoRA: Orthogonal Projection LoRA Prevents Catastrophic Forgetting during Parameter-Efficient Fine-Tuning
• REPAIR: Robust Editing via Progressive Adaptive Intervention and Reintegration
• Dynamic Orthogonal Continual Fine-tuning for Mitigating Catastrophic Forgettings
• Understanding Catastrophic Interference: On the Identifibility of Latent Representations
• Task-Agnostic Federated Continual Learning via Replay-Free Gradient Projection
• Null-Space Filtering for Data-Free Continual Model Merging: Preserving Transparency, Promoting Fidelity
• Efficient Reinforcement Learning by Reducing Forgetting with Elephant Activation Functions
• CL²GEC: A Multi-Discipline Benchmark for Continual Learning in Chinese Literature Grammatical Error Correction
• ONG: Orthogonal Natural Gradient Descent
• Continual Learning with Deep Artificial Neurons
• Linear Mode Connectivity in Multitask and Continual Learning
• C-Flat++: Towards a More Efficient and Powerful Framework for Continual Learning
• Adaptive Variance-Penalized Continual Learning with Fisher Regularization
• Statistical Theory of Multi-stage Newton Iteration Algorithm for Online Continual Learning
• Lifelong Learner: Discovering Versatile Neural Solvers for Vehicle Routing Problems
• Routing Networks with Co-training for Continual Learning
• MLLM-CTBench: A Benchmark for Continual Instruction Tuning with Reasoning Process Diagnosis
• Modular Delta Merging with Orthogonal Constraints: A Scalable Framework for Continual and Reversible Model Composition
• Continual Speaker Identity Unlearning with Minimal Interference

Related

• Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent
• Layerwise Optimization by Gradient Decomposition for Continual Learning
• Restricted Orthogonal Gradient Projection for Continual Learning
• Defeating Catastrophic Forgetting via Enhanced Orthogonal Weights Modification
• Gradient-Free Continual Learning