Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspective

2025/05/27 by JoonKyu Kim, Yejin Kim, Kim, Joonkyu +3 · 1 citation
Computer Science · Neuroscience · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Memory Processes and Influences #Visual Attention and Saliency Detection

paper · pdf · doi:10.48550/arxiv.2505.20970

openalex publication_date 2025/05/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In continual learning scenarios, catastrophic forgetting of previously learned tasks is a critical issue, making it essential to effectively measure such forgetting. Recently, there has been growing interest in focusing on representation forgetting, the forgetting measured at the hidden layer. In this paper, we provide the first theoretical analysis of representation forgetting and use this analysis to better understand the behavior of continual learning. First, we introduce a new metric called representation discrepancy, which measures the difference between representation spaces constructed by two snapshots of a model trained through continual learning. We demonstrate that our proposed metric serves as an effective surrogate for the representation forgetting while remaining analytically tractable. Second, through mathematical analysis of our metric, we derive several key findings about the dynamics of representation forgetting: the forgetting occurs more rapidly to a higher degree as the layer index increases, while increasing the width of the network slows down the forgetting process. Third, we support our theoretical findings through experiments on real image datasets, including Split-CIFAR100 and ImageNet1K.

Citations

• A Statistical Theory of Regularization-Based Continual Learning
• Provable Contrastive Continual Learning
• Understanding Forgetting in Continual Learning with Linear Regression
• Integrating Present and Past in Unsupervised Continual Learning
• On the Diminishing Returns of Width for Continual Learning
• Investigating Forgetting in Pre-Trained Representations Through Continual Learning
• The Ideal Continual Learner: An Agent That Never Forgets
• Fixed Design Analysis of Regularization-Based Continual Learning
• Theory on Forgetting and Generalization of Continual Learning
• A Theoretical Study on Solving Continual Learning
• Feature Forgetting in Continual Representation Learning
• Theoretical Understanding of the Information Flow on Continual Learning Performance
• Memory Bounds for Continual Learning
• Probing Representation Forgetting in Supervised and Unsupervised Continual Learning
• Provable and Efficient Continual Representation Learning
• Provable Continual Learning via Sketched Jacobian Approximations
• Self-Supervised Models are Continual Learners
• Provable Lifelong Learning of Representations
• New Insights on Reducing Abrupt Representation Change in Online Continual Learning
• New Insights on Reducing Abrupt Representation Change in Online Continual Learning
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics
• Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent
• Continual Unsupervised Representation Learning
• Similarity of Neural Network Representations Revisited
• Stronger generalization bounds for deep nets via a compression approach
• A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
• Deep Residual Learning for Image Recognition
• A PAC-Bayesian bound for Lifelong Learning
• Domain Adaptation: Learning Bounds and Algorithms

Cited by

• Forgetting is Everywhere

Related

• Probing Representation Forgetting in Supervised and Unsupervised Continual Learning
• Feature Forgetting in Continual Representation Learning
• Continual Learning by Modeling Intra-Class Variation
• On robustness of generative representations against catastrophic forgetting
• Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics