A Simple Baseline for Stable and Plastic Neural Networks

2025/07/14 by Eberhard Künzel, Künzel, Étienne, Achref Jaziri +3
Computer Science · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Face recognition and analysis #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2507.10637

openalex publication_date 2025/07/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Continual learning in computer vision requires that models adapt to a continuous stream of tasks without forgetting prior knowledge, yet existing approaches often tip the balance heavily toward either plasticity or stability. We introduce RDBP, a simple, low-overhead baseline that unites two complementary mechanisms: ReLUDown, a lightweight activation modification that preserves feature sensitivity while preventing neuron dormancy, and Decreasing Backpropagation, a biologically inspired gradient-scheduling scheme that progressively shields early layers from catastrophic updates. Evaluated on the Continual ImageNet benchmark, RDBP matches or exceeds the plasticity and stability of state-of-the-art methods while reducing computational cost. RDBP thus provides both a practical solution for real-world continual learning and a clear benchmark against which future continual learning strategies can be measured.

Citations

• Plasticity-Aware Mixture of Experts for Learning Under QoE Shifts in Adaptive Video Streaming
• Neuroplastic Expansion in Deep Reinforcement Learning
• Mitigating the Stability-Plasticity Dilemma in Adaptive Train Scheduling with Curriculum-Driven Continual DQN Expansion
• Slow and Steady Wins the Race: Maintaining Plasticity with Hare and Tortoise Networks
• Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning
• Disentangling the Causes of Plasticity Loss in Neural Networks
• On the Stability-Plasticity Dilemma of Class-Incremental Learning
• The Dormant Neuron Phenomenon in Deep Reinforcement Learning
• New Insights for the Stability-Plasticity Dilemma in Online Continual Learning
• Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability
• Fine-tuned Language Models are Continual Learners
• A Closer Look at Rehearsal-Free Continual Learning
• A study on the plasticity of neural networks
• Adaptive Rational Activations to Boost Deep Reinforcement Learning
• Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference
• Progress & Compress: A scalable framework for continual learning
• Lifelong Learning with Dynamically Expandable Networks
• Continual Learning with Deep Generative Replay
• How far can we go without convolution: Improving fully-connected networks
• A Survey on Transfer Learning

Related

• Maintaining Plasticity in Deep Continual Learning
• Simpler is Better: off-the-shelf Continual Learning Through Pretrained Backbones
• Continual Learning with Pretrained Backbones by Tuning in the Input Space
• Generalizable Two-Branch Framework for Image Class-Incremental Learning
• Learning a Low-Rank Feature Representation: Achieving Better Trade-Off between Stability and Plasticity in Continual Learning