Linear Mode Connectivity in Multitask and Continual Learning

2020/10/09 by Seyed Iman Mirzadeh, Mirzadeh, Seyed Iman, Mehrdad Farajtabar +7 · 12 citations
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Sparse and Compressive Sensing Techniques

paper · pdf · doi:10.48550/arxiv.2010.04495

openalex publication_date 2020/10/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Continual (sequential) training and multitask (simultaneous) training are often attempting to solve the same overall objective: to find a solution that performs well on all considered tasks. The main difference is in the training regimes, where continual learning can only have access to one task at a time, which for neural networks typically leads to catastrophic forgetting. That is, the solution found for a subsequent task does not perform well on the previous ones anymore. However, the relationship between the different minima that the two training regimes arrive at is not well understood. What sets them apart? Is there a local structure that could explain the difference in performance achieved by the two different schemes? Motivated by recent work showing that different minima of the same task are typically connected by very simple curves of low error, we investigate whether multitask and continual solutions are similarly connected. We empirically find that indeed such connectivity can be reliably achieved and, more interestingly, it can be done by a linear path, conditioned on having the same initialization for both. We thoroughly analyze this observation and discuss its significance for the continual learning process. Furthermore, we exploit this finding to propose an effective algorithm that constrains the sequentially learned minima to behave as the multitask solution. We show that our method outperforms several state of the art continual learning algorithms on various vision benchmarks.

Citations

• Catastrophic Forgetting, Rehearsal and Pseudorehearsal
• Overcoming Catastrophic Forgetting by Incremental Moment Matching
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Experience Replay for Continual Learning
• Functional Regularisation for Continual Learning with Gaussian Processes
• On Tiny Episodic Memories in Continual Learning
• An Empirical Study of Example Forgetting during Deep Neural Network Learning
• Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference
• Measuring Catastrophic Forgetting in Neural Networks
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Orthogonal Gradient Descent for Continual Learning
• Variational Prototype Replays for Continual Learning

Cited by

• Rethinking Expert Training for Model Merging with Prompt Learning
• Sequencing to Mitigate Catastrophic Forgetting in Continual Learning
• Learning with Preserving for Continual Multitask Learning
• Model Merging Improves Zero-Shot Generalization in Bioacoustic Foundation Models
• Linear Mode Connectivity under Data Shifts for Deep Ensembles of Image Classifiers
• More Than Memory Savings: Zeroth-Order Optimization Mitigates Forgetting in Continual Learning
• FedGTEA: Federated Class-Incremental Learning with Gaussian Task Embedding and Alignment
• Exploring and Leveraging Class Vectors for Classifier Editing
• Robustness and Regularization in Hierarchical Re-Basin
• Closing the Oracle Gap: Increment Vector Transformation for Class Incremental Learning
• Null-Space Filtering for Data-Free Continual Model Merging: Preserving Transparency, Promoting Fidelity
• Forgetting of task-specific knowledge in model merging-based continual learning

Related

• Optimal Task Order for Continual Learning of Multiple Tasks
• Residual Continual Learning
• Is Multi-Task Learning an Upper Bound for Continual Learning?
• Does Continual Learning Equally Forget All Parameters?
• Task Agnostic Continual Learning via Meta Learning