An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks

2013/12/21 by Ian Goodfellow, Ian J. Goodfellow, Goodfellow, Ian J. +9 · 165 citations
Computer Science · Mathematics · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Multimodal Machine Learning Applications #Neural and Evolutionary Computing (cs.NE) #cs.LG #cs.NE #stat.ML

paper · pdf · doi:10.48550/arxiv.1312.6211

openalex publication_date 2013/12/21 · arxiv created 2015/03/04 · arxiv updated 2015/03/05 · openalex created_date 2016/06/24 · openalex updated_date 2026/07/28

Abstract

Catastrophic forgetting is a problem faced by many machine learning models and algorithms. When trained on one task, then trained on a second task, many machine learning models "forget" how to perform the first task. This is widely believed to be a serious problem for neural networks. Here, we investigate the extent to which the catastrophic forgetting problem occurs for modern neural networks, comparing both established and recent gradient-based training algorithms and activation functions. We also examine the effect of the relationship between the first task and the second task on catastrophic forgetting. We find that it is always best to train using the dropout algorithm--the dropout algorithm is consistently best at adapting to the new task, remembering the old task, and has the best tradeoff curve between these two extremes. We find that different tasks and relationships between tasks result in very different rankings of activation function performance. This suggests the choice of activation function should always be cross-validated.

Citations

• Improving neural networks by preventing co-adaptation of feature detectors
• Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions.

Cited by

• Certifying the Right to Be Forgotten: Primal-Dual Optimization for Sample and Label Unlearning in Vertical Federated Learning
• Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning
• Breaking the Synthetic-Real Domain Shortcut for Training-Free Generative Replay-based Class Incremental Learning
• PIcsC: Partitioning-Induced Covariate Shift Correction
• The Erasure Illusion: Stress-Testing the Generalization of LLM Forgetting Evaluation
• AL-GNN: Privacy-Preserving and Replay-Free Continual Graph Learning via Analytic Learning
• Distillation-Guided Structural Transfer for Continual Learning Beyond Sparse Distributed Memory
• CIP-Net: Continual Interpretable Prototype-based Network
• Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updates
• Knowledge Distillation for Continual Learning of Biomedical Neural Fields
• Mitigating Catastrophic Forgetting in Streaming Generative and Predictive Learning via Stateful Replay
• Intrinsic preservation of plasticity in continual quantum learning
• Dynamic Participation in Federated Learning: Benchmarks and a Knowledge Pool Plugin
• Donors and Recipients: On Asymmetric Transfer Across Tasks and Languages with Parameter-Efficient Fine-Tuning
• Gene Incremental Learning for Single-Cell Transcriptomics
• Data Heterogeneity and Forgotten Labels in Split Federated Learning
• Mitigating Negative Flips via Margin Preserving Training
• FLEX: Continuous Agent Evolution via Forward Learning from Experience
• A Representation Sharpening Framework for Zero Shot Dense Retrieval
• Progressive Multimodal Alignment for Continual Instruction Tuning
• Explaining Robustness to Catastrophic Forgetting Through Incremental Concept Formation
• COLA: Continual Learning via Autoencoder Retrieval of Adapters
• Online Learning Defense against Iterative Jailbreak Attacks via Prompt Optimization
• Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences
• On the Theory of Continual Learning with Gradient Descent for Neural Networks
• Unlearning in Diffusion models under Data Constraints: A Variational Inference Approach
• MACE: A Hybrid LLM Serving System with Colocated SLO-aware Continuous Retraining Alignment
• Cross-Domain Multi-Person Human Activity Recognition via Near-Field Wi-Fi Sensing
• Activation Function Design Sustains Plasticity in Continual Learning
• Spectral Collapse Drives Loss of Plasticity in Deep Continual Learning
• ONG: Orthogonal Natural Gradient Descent
• Less-forgetful Learning for Domain Expansion in Deep Neural Networks
• Variational Federated Multi-Task Learning
• Using Contrastive Learning to Improve Two-Way Reasoning in Large Language Models: The Obfuscation Task as a Case Study
• Selfless Sequential Learning
• Positive-Congruent Training: Towards Regression-Free Model Updates
• FedAT: A High-Performance and Communication-Efficient Federated Learning System with Asynchronous Tiers
• Revisiting Deepfake Detection: Chronological Continual Learning and the Limits of Generalization
• Collaborative Semantic Aggregation and Calibration for Federated Domain Generalization
• Behavior From the Void: Unsupervised Active Pre-Training
• Continual Learning with Self-Organizing Maps
• Unlearning Comparator: A Visual Analytics System for Comparative Evaluation of Machine Unlearning Methods
• Adaptive Variance-Penalized Continual Learning with Fisher Regularization
• Index-Aligned Query Distillation for Transformer-based Incremental Object Detection
• Two-Level Residual Distillation based Triple Network for Incremental Object Detection
• ESSENTIAL: Episodic and Semantic Memory Integration for Video Class-Incremental Learning
• Training Binary Neural Networks using the Bayesian Learning Rule
• Wide Neural Networks Forget Less Catastrophically
• Continual Learning in Neural Networks
• Dissecting Catastrophic Forgetting in Continual Learning by Deep Visualization
• Separation and Collaboration: Two-Level Routing Grouped Mixture-of-Experts for Multi-Domain Continual Learning
• Prompt Tuning for Few-Shot Continual Learning Named Entity Recognition
• Contrast R-CNN for Continual Learning in Object Detection
• Comparing Knowledge Injection Methods for LLMs in a Low-Resource Regime
• A Study on Regularization-Based Continual Learning Methods for Indic ASR
• Learning from Oblivion: Predicting Knowledge Overflowed Weights via Retrodiction of Forgetting
• Prediction-Oriented Subsampling from Data Streams
• Exploring Stability-Plasticity Trade-offs for Continual Named Entity Recognition
• Reinitializing weights vs units for maintaining plasticity in neural networks
• Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics
• LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systems
• Revisiting Catastrophic Forgetting in Class Incremental Learning
• Multi-Attribute Selectivity Estimation Using Deep Learning
• Graph-Based Continual Learning
• Generative replay with feedback connections as a general strategy for continual learning
• Regularizing Towards Permutation Invariance in Recurrent Models
• iLGaCo: Incremental Learning of Gait Covariate Factors
• Large-scale Kernel Methods and Applications to Lifelong Robot Learning
• Three scenarios for continual learning
• Swin-TUNA : A Novel PEFT Approach for Accurate Food Image Segmentation
• Information-Theoretic Generalization Bounds of Replay-based Continual Learning
• Off-Policy Deep Reinforcement Learning without Exploration
• Online Continual Learning in Image Classification: An Empirical Survey
• The unreasonable effectiveness of Batch-Norm statistics in addressing catastrophic forgetting across medical institutions
• CLA: Latent Alignment for Online Continual Self-Supervised Learning
• SeNA-CNN: Overcoming Catastrophic Forgetting in Convolutional Neural Networks by Selective Network Augmentation
• Overcoming catastrophic forgetting in neural networks
• The Bayesian Approach to Continual Learning: An Overview
• A statistical physics framework for optimal learning
• Applying Incremental Deep Neural Networks-based Posture Recognition Model for Injury Risk Assessment in Construction
• Temporal Information Retrieval via Time-Specifier Model Merging
• NeuroSynth: A Biologically Inspired Continual Reinforcement Learning Architecture for Mitigating Catastrophic Forgetting
• DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment
• Posterior Meta-Replay for Continual Learning
• Continual Learning Using Bayesian Neural Networks
• NIRANTAR: Continual Learning with New Languages and Domains on Real-world Speech Data
• BooVAE: Boosting Approach for Continual Learning of VAE
• FedRef: Communication-Efficient Bayesian Fine-Tuning using a Reference Model
• Continual Learning Through Synaptic Intelligence
• Consistency-Driven Calibration and Matching for Few-Shot Class-Incremental Learning
• Dealing with the Evil Twins: Improving Random Augmentation by Addressing Catastrophic Forgetting of Diverse Augmentations
• MEMOIR: Lifelong Model Editing with Minimal Overwrite and Informed Retention for LLMs
• Toward Understanding Catastrophic Forgetting in Continual Learning
• Continual Learning with Columnar Spiking Neural Networks
• Compliant Residual DAgger: Improving Real-World Contact-Rich Manipulation with Human Corrections
• Mr. Snuffleupagus at SemEval-2025 Task 4: Unlearning Factual Knowledge from LLMs Using Adaptive RMU
• Encoders and Ensembles for Task-Free Continual Learning
• Adapting Whisper for Parameter-efficient Code-Switching Speech Recognition via Soft Prompt Tuning
• Neural Machine Translation: A Review and Survey
• Dynamic Context-oriented Decomposition for Task-aware Low-rank Adaptation with Less Forgetting and Faster Convergence
• Lifelong Object Detection
• Solving hybrid machine learning tasks by traversing weight space geodesics
• Certified Unlearning for Neural Networks
• Problems in AI research and how the SP System may help to solve them
• Continual Classification Learning Using Generative Models
• Hierarchical-Task-Aware Multi-modal Mixture of Incremental LoRA Experts for Embodied Continual Learning
• Unpacking Information Bottlenecks: Unifying Information-Theoretic Objectives in Deep Learning
• A Review of Computer Vision Methods in Network Security
• Localizing Catastrophic Forgetting in Neural Networks
• Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspective
• Adapt before Continual Learning
• Towards Recognizing New Semantic Concepts in New Visual Domains
• Powerpropagation: A sparsity inducing weight reparameterisation
• Deep Generative Dual Memory Network for Continual Learning
• Towards Robust Evaluations of Continual Learning
• AVROBUSTBENCH: Benchmarking the Robustness of Audio-Visual Recognition Models at Test-Time
• Enhancing Multimodal Continual Instruction Tuning with BranchLoRA
• Towards Minimizing Feature Drift in Model Merging: Layer-wise Task Vector Fusion for Adaptive Knowledge Integration
• SC-LoRA: Balancing Efficient Fine-tuning and Knowledge Preservation via Subspace-Constrained LoRA
• Information-Theoretic Complementary Prompts for Improved Continual Text Classification
• Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
• Speech-IFEval: Evaluating Instruction-Following and Quantifying Catastrophic Forgetting in Speech-Aware Language Models
• Continual Learning via Online Leverage Score Sampling
• A good body is all you need: avoiding catastrophic interference via agent architecture search
• Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Models
• Preserving Earlier Knowledge in Continual Learning with the Help of All Previous Feature Extractors
• Locate-then-Merge: Neuron-Level Parameter Fusion for Mitigating Catastrophic Forgetting in Multimodal LLMs
• Joint Flashback Adaptation for Forgetting-Resistant Instruction Tuning
• Multi-Modality Expansion and Retention for LLMs through Parameter Merging and Decoupling
• Domain Adaptive Medical Image Segmentation via Adversarial Learning of Disease-Specific Spatial Patterns
• AdaHAT: Adaptive Hard Attention to the Task in Task-Incremental Learning
• Data augmentation and pre-trained networks for extremely low data regimes unsupervised visual inspection
• The LMU Munich System for the WMT 2020 Unsupervised Machine Translation Shared Task
• Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theory
• OpenLORIS-Object: A Robotic Vision Dataset and Benchmark for Lifelong Deep Learning
• CL-CaGAN: Capsule differential adversarial continuous learning for cross-domain hyperspectral anomaly detection
• Ready2Unlearn: A Learning-Time Approach for Preparing Models with Future Unlearning Readiness
• A Foliated View of Transfer Learning
• Low-Complexity Inference in Continual Learning via Compressed Knowledge Transfer
• Federated Learning with Additional Mechanisms on Clients to Reduce Communication Costs
• (How) Learning Rates Regulate Catastrophic Overtraining
• Unsupervised Transfer Learning with Self-Supervised Remedy
• Incremental Class Learning using Variational Autoencoders with Similarity Learning
• Defining Benchmarks for Continual Few-Shot Learning
• Impact of transfer learning methods and dataset characteristics on generalization in birdsong classification
• Adaptive Compression-based Lifelong Learning
• Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learning
• Organizing Experience: A Deeper Look at Replay Mechanisms for Sample-based Planning in Continuous State Domains
• Differentially Private Distributed Learning for Language Modeling Tasks
• Concept-Oriented Deep Learning
• Iterative Semi-parametric Dynamics Model Learning For Autonomous Racing
• Med-HEAL: Analyzing and Mitigating Hallucinations in Medical LLMs with Hallucination-Aware In-Context Learning
• Personalized Artificial General Intelligence (AGI) via Neuroscience-Inspired Continuous Learning Systems
• Transferring Autonomous Driving Knowledge on Simulated and Real\n Intersections
• Debiasing LLMs by Fine-tuning
• Unifying Regularisation Methods for Continual Learning
• Adaptive Intrusion Detection System using Transformer-Based Neural Networks and Continual Learning Approach with Adversarial Investigation
• Meta-learnt priors slow down catastrophic forgetting in neural networks
• Catastrophic Forgetting: Still a Problem for DNNs
• Semi-parametric Memory Consolidation: Towards Brain-like Deep Continual Learning
• SelfHAR
• Bayesian continual learning and forgetting in neural networks
• Memory-Statistics Tradeoff in Continual Learning with Structural Regularization
• Balancing Stability and Plasticity in Pretrained Detector: A Dual-Path Framework for Incremental Object Detection
• Knowledge-Instruct: Effective Continual Pre-training from Limited Data using Instructions

Related

• Does the Adam Optimizer Exacerbate Catastrophic Forgetting?
• Understanding the Role of Training Regimes in Continual Learning
• Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy
• Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics
• Localizing Catastrophic Forgetting in Neural Networks