Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics

2020/07/14 by Vinay Ramasesh, Ethan Dyer, Ramasesh, Vinay V. +3 · 5 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2007.07400

openalex publication_date 2020/07/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

A central challenge in developing versatile machine learning systems is catastrophic forgetting: a model trained on tasks in sequence will suffer significant performance drops on earlier tasks. Despite the ubiquity of catastrophic forgetting, there is limited understanding of the underlying process and its causes. In this paper, we address this important knowledge gap, investigating how forgetting affects representations in neural network models. Through representational analysis techniques, we find that deeper layers are disproportionately the source of forgetting. Supporting this, a study of methods to mitigate forgetting illustrates that they act to stabilize deeper layers. These insights enable the development of an analytic argument and empirical picture relating the degree of forgetting to representational similarity between tasks. Consistent with this picture, we observe maximal forgetting occurs for task sequences with intermediate similarity. We perform empirical studies on the standard split CIFAR-10 setup and also introduce a novel CIFAR-100 based task approximating realistic input distribution shift.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions.
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• Learning without Forgetting
• Overcoming Catastrophic Forgetting by Incremental Moment Matching
• mixup: Beyond Empirical Risk Minimization
• Continual Learning via Neural Pruning
• Deep Residual Learning for Image Recognition
• Measuring Catastrophic Forgetting in Neural Networks
• Affinity and Diversity: Quantifying Mechanisms of Data Augmentation
• A Survey of Deep Learning for Scientific Discovery

Cited by

• Heads collapse, features stay: Why Replay needs big buffers
• Data Heterogeneity and Forgotten Labels in Split Federated Learning
• OPLoRA: Orthogonal Projection LoRA Prevents Catastrophic Forgetting during Parameter-Efficient Fine-Tuning
• Continual Learning with Query-Only Attention
• AIM: Adaptive Intra-Network Modulation for Balanced Multimodal Learning

Related

• Negotiated Representations to Prevent Forgetting in Machine Learning Applications
• Probing Representation Forgetting in Supervised and Unsupervised Continual Learning
• Diagnosing Catastrophe: Large parts of accuracy loss in continual learning can be accounted for by readout misalignment
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• On robustness of generative representations against catastrophic forgetting