Distillation-Guided Structural Transfer for Continual Learning Beyond Sparse Distributed Memory

2025/12/17 by Xue, Huiyan, Ran, Xuming, Li, Yaxin +4
Computer Science · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2512.15267

openalex publication_date 2025/12/17 · openalex created_date 2025/12/19 · openalex updated_date 2026/07/28

Abstract

Sparse neural systems are gaining traction for efficient continual learning due to their modularity and low interference. Architectures such as Sparse Distributed Memory Multi-Layer Perceptrons (SDMLP) construct task-specific subnetworks via Top-K activation and have shown resilience against catastrophic forgetting. However, their rigid modularity limits cross-task knowledge reuse and leads to performance degradation under high sparsity. We propose Selective Subnetwork Distillation (SSD), a structurally guided continual learning framework that treats distillation not as a regularizer but as a topology-aligned information conduit. SSD identifies neurons with high activation frequency and selectively distills knowledge within previous Top-K subnetworks and output logits, without requiring replay or task labels. This enables structural realignment while preserving sparse modularity. Experiments on Split CIFAR-10, CIFAR-100, and MNIST demonstrate that SSD improves accuracy, retention, and representation coverage, offering a structurally grounded solution for sparse continual learning.

Citations

• Brain-inspired continual pre-trained learner via silent synaptic consolidation
• SparCL: Sparse Continual Learning on the Edge
• Task Agnostic Representation Consolidation: a Self-supervised based Continual Learning Approach
• Mixture-of-Experts with Expert Choice Routing
• Training Neural Networks with Fixed Sparse Masks
• PLOP: Learning without Forgetting for Continual Semantic Segmentation
• Supermasks in Superposition
• Contrastive Representation Distillation
• Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization
• Learning without Memorizing
• Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
• Towards Robust Evaluations of Continual Learning
• Progress & Compress: A scalable framework for continual learning
• Reinforced Continual Learning
• On the importance of single directions for generalization
• Piggyback: Adapting a Single Network to Multiple Tasks by Learning to\n Mask Weights
• PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning
• Incremental Learning of Object Detectors without Catastrophic Forgetting
• Gradient Episodic Memory for Continual Learning
• Continual Learning with Deep Generative Replay
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• iCaRL: Incremental Classifier and Representation Learning
• Learning without Forgetting
• Distilling the Knowledge in a Neural Network
• Striving for Simplicity: The All Convolutional Net
• ImageNet Large Scale Visual Recognition Challenge
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks

Related

• Dynamically Modular and Sparse General Continual Learning
• Subspace Distillation for Continual Learning
• Mastering Continual Reinforcement Learning through Fine-Grained Sparse Network Allocation and Dormant Neuron Exploration
• Bio-Inspired, Task-Free Continual Learning through Activity Regularization
• Continual learning under domain transfer with sparse synaptic bursting