On Understanding of the Dynamics of Model Capacity in Continual Learning

2025/08/11 by Supriyo Chakraborty, Krishnan Raghavan, Chakraborty, Supriyo +1
Computer Science · Medicine · #Domain Adaptation and Few-Shot Learning #Artificial Intelligence in Healthcare and Education #Advanced Graph Neural Networks

paper · pdf · doi:10.48550/arxiv.2508.08052

Abstract

The stability-plasticity dilemma, closely related to a neural network's (NN) capacity-its ability to represent tasks-is a fundamental challenge in continual learning (CL). Within this context, we introduce CL's effective model capacity (CLEMC) that characterizes the dynamic behavior of the stability-plasticity balance point. We develop a difference equation to model the evolution of the interplay between the NN, task data, and optimization procedure. We then leverage CLEMC to demonstrate that the effective capacity-and, by extension, the stability-plasticity balance point is inherently non-stationary. We show that regardless of the NN architecture or optimization method, a NN's ability to represent new tasks diminishes when incoming task distributions differ from previous ones. We conduct extensive experiments to support our theoretical findings, spanning a range of architectures-from small feedforward network and convolutional networks to medium-sized graph neural networks and transformer-based large language models with millions of parameters.

Citations

• The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting -- An Analytical Model
• Effect of Optimizer, Initializer, and Architecture of Hypernetworks on Continual Learning from Demonstration
• Improving Plasticity in Online Continual Learning via Collaborative Learning
• Adversarial Robust Memory-Based Continual Learner
• RTRA: Rapid Training of Regularization-based Approaches in Continual Learning
• Density Distribution-based Learning Framework for Addressing Online Continual Learning Challenges
• Elephant Neural Networks: Born to Be a Continual Learner
• Adaptive Reorganization of Neural Pathways for Continual Learning with Spiking Neural Networks
• Continual Learning with Dirichlet Generative-based Rehearsal
• GRASP: A Rehearsal Policy for Efficient Online Continual Learning
• Maintaining Plasticity in Continual Learning via Regenerative Regularization
• Efficient Model Adaptation for Continual Learning at the Edge
• Learnability and Algorithm for Continual Learning
• Continual Learning in Linear Classification on Separable Data
• Overcoming the Stability Gap in Continual Learning
• Investigating Forgetting in Pre-Trained Representations Through Continual Learning
• On the Effectiveness of Equivariant Regularization for Robust Online Continual Learning
• PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
• Open-World Continual Learning: Unifying Novelty Detection and Continual Learning
• Theory on Forgetting and Generalization of Continual Learning
• A Comprehensive Survey of Continual Learning: Theory, Method and Application
• Analysis of Catastrophic Forgetting for Random Orthogonal Transformation Tasks in the Overparameterized Regime
• Continual evaluation for lifelong learning: Identifying the stability gap
• Wide Neural Networks Forget Less Catastrophically
• Formalizing the Generalization-Forgetting Trade-off in Continual Learning
• Avalanche: an End-to-End Library for Continual Learning
• Understanding the Role of Training Regimes in Continual Learning
• Learning to Continually Learn
• Deep Double Descent: Where Bigger Models and More Data Hurt
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Meta-Learning Representations for Continual Learning
• Fast Graph Representation Learning with PyTorch Geometric

Related

• Achieving a Better Stability-Plasticity Trade-off via Auxiliary Networks in Continual Learning
• New metrics for analyzing continual learners
• A study on the plasticity of neural networks
• Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspective
• Keep Moving: identifying task-relevant subspaces to maximise plasticity for newly learned tasks