Merge and Bound: Direct Manipulations on Weights for Class Incremental Learning

2025/11/26 by Taehoon Kim, Kim, Taehoon, Dong-Hwan Jang +3
Computer Science · #Advanced Graph Neural Networks #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Human Pose and Action Recognition #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2511.21490

openalex publication_date 2025/11/26 · openalex created_date 2025/11/28 · openalex updated_date 2026/07/28

Abstract

We present a novel training approach, named Merge-and-Bound (M&B) for Class Incremental Learning (CIL), which directly manipulates model weights in the parameter space for optimization. Our algorithm involves two types of weight merging: inter-task weight merging and intra-task weight merging. Inter-task weight merging unifies previous models by averaging the weights of models from all previous stages. On the other hand, intra-task weight merging facilitates the learning of current task by combining the model parameters within current stage. For reliable weight merging, we also propose a bounded update technique that aims to optimize the target model with minimal cumulative updates and preserve knowledge from previous tasks; this strategy reveals that it is possible to effectively obtain new models near old ones, reducing catastrophic forgetting. M&B is seamlessly integrated into existing CIL methods without modifying architecture components or revising learning objectives. We extensively evaluate our algorithm on standard CIL benchmarks and demonstrate superior performance compared to state-of-the-art methods.

Citations

• Surgical Fine-Tuning Improves Adaptation to Distribution Shifts
• Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation
• Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
• Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution
• Robust fine-tuning of zero-shot models
• On Learning the Geodesic Path for Incremental Learning
• Learning Transferable Visual Models From Natural Language Supervision
• Denoising Diffusion Probabilistic Models
• Language Models are Few-Shot Learners
• Generative Feature Replay For Class-Incremental Learning
• Distance-Based Regularisation of Deep Networks for Fine-Tuning
• Large Scale Incremental Learning
• Similarity of Neural Network Representations Revisited
• Three scenarios for continual learning
• Learning to Remember: A Synaptic Plasticity Driven Framework for\n Continual Learning
• Overcoming Catastrophic Forgetting with Unlabeled Data in the Wild
• Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
• Averaging Weights Leads to Wider Optima and Better Generalization
• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
• Memory Aware Synapses: Learning what (not) to forget
• Continual Learning with Deep Generative Replay
• iCaRL: Incremental Classifier and Representation Learning
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Progressive Neural Networks
• Distilling the Knowledge in a Neural Network
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• FitNets: Hints for Thin Deep Nets
• ImageNet Large Scale Visual Recognition Challenge
• ImageNet Large Scale Visual Recognition Challenge

Related

• Adaptive Weighted Parameter Fusion with CLIP for Class-Incremental Learning
• Mixture of Noise for Pre-Trained Model-Based Class-Incremental Learning
• SS-IL: Separated Softmax for Incremental Learning
• Joint Input and Output Coordination for Class-Incremental Learning
• MOS: Model Surgery for Pre-Trained Model-Based Class-Incremental Learning