Lifelong Learning with Dynamically Expandable Networks

2017/08/04 by Jaehong Yoon, Yoon, Jaehong, Eunho Yang +5 · 50 citations
Computer Science · Medicine · #Advanced Neural Network Applications #COVID-19 diagnosis using AI #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #I.2.10 #I.2.6 #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.1708.01547

openalex publication_date 2017/08/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose a novel deep network architecture for lifelong learning which we refer to as Dynamically Expandable Network (DEN), that can dynamically decide its network capacity as it trains on a sequence of tasks, to learn a compact overlapping knowledge sharing structure among tasks. DEN is efficiently trained in an online manner by performing selective retraining, dynamically expands network capacity upon arrival of each task with only the necessary number of units, and effectively prevents semantic drift by splitting/duplicating units and timestamping them. We validate DEN on multiple public datasets under lifelong learning scenarios, on which it not only significantly outperforms existing lifelong learning methods for deep networks, but also achieves the same level of performance as the batch counterparts with substantially fewer number of parameters. Further, the obtained network fine-tuned on all tasks obtained significantly better performance over the batch models, which shows that it can be used to estimate the optimal network structure even when all tasks are available in the first place.

Citations

• Gradient-based learning applied to document recognition
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Learning continually with representational drift
• Scalable Class-Incremental Learning Based on Parametric Neural Collapse
• LibContinual: A Comprehensive Library towards Realistic Continual Learning
• Sequencing to Mitigate Catastrophic Forgetting in Continual Learning
• Self-Motivated Growing Neural Network for Adaptive Architecture via Local Structural Plasticity
• Task-Aware Multi-Expert Architecture For Lifelong Deep Learning
• Multi-Generator Continual Learning for Robust Delay Prediction in 6G
• Provably Safe Model Updates
• Neuroscience-Inspired Memory Replay for Continual Learning: A Comparative Study of Predictive Coding and Backpropagation-Based Strategies
• Probabilistic Hash Embeddings for Online Learning of Categorical Features
• Continual Alignment for SAM: Rethinking Foundation Models for Medical Image Segmentation in Continual Learning
• Catastrophic Forgetting in Kolmogorov-Arnold Networks
• Neuro-Logic Lifelong Learning
• Retrofit: Continual Learning with Bounded Forgetting for Security Applications
• FSC-Net: Fast-Slow Consolidation Networks for Continual Learning
• Clustering-Based Weight Orthogonalization for Stabilizing Deep Reinforcement Learning
• SAMix: Calibrated and Accurate Continual Learning via Sphere-Adaptive Mixup and Neural Collapse
• Continual Knowledge Consolidation LORA for Domain Incremental Learning
• COLA: Continual Learning via Autoencoder Retrieval of Adapters
• KORE: Enhancing Knowledge Injection for Large Multimodal Models via Knowledge-Oriented Controls
• Source-Free Cross-Domain Continual Learning
• Diffusion-Classifier Synergy: Reward-Aligned Learning via Mutual Boosting Loop for FSCIL
• Deep Generative Continual Learning using Functional LoRA: FunLoRA
• Diagnosing Shortcut-Induced Rigidity in Continual Learning: The Einstellung Rigidity Index (ERI)
• A Computational Perspective on NeuroAI and Synthetic Biological Intelligence
• Policy Compatible Skill Incremental Learning via Lazy Learning Interface
• DECODE: Tackling Representation and Decision Degradation in Continual AI-Generated Image Detection
• Shared-Weights Extender and Gradient Voting for Neural Network Expansion
• COLT: Enhancing Video Large Language Models with Continual Tool Usage
• Efficient Reinforcement Learning by Reducing Forgetting with Elephant Activation Functions
• An Unlearning Framework for Continual Learning
• Unbiased Online Curvature Approximation for Regularized Graph Continual Learning
• The Lifecycle Principle: Stabilizing Dynamic Neural Networks with State Memory
• ONG: Orthogonal Natural Gradient Descent
• Single-Net Continual Learning with Progressive Segmented Training (PST)
• Unsupervised Identification and Replay-based Detection (UIRD) for New Category Anomaly Detection in ECG Signal
• Learning Yourself: Class-Incremental Semantic Segmentation with Language-Inspired Bootstrapped Disentanglement
• LIRA: Lifelong Image Restoration from Unknown Blended Distortions
• BRAIN: Bias-Mitigation Continual Learning Approach to Vision-Brain Understanding
• Continual learning using hash-routed convolutional neural networks
• Index-Aligned Query Distillation for Transformer-based Incremental Object Detection
• Memorisation and forgetting in a learning Hopfield neural network: bifurcation mechanisms, attractors and basins
• Hierarchical Visual Prompt Learning for Continual Video Instance Segmentation
• Statistical Theory of Multi-stage Newton Iteration Algorithm for Online Continual Learning
• Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting
• Exploring Stability-Plasticity Trade-offs for Continual Named Entity Recognition
• RainbowPrompt: Diversity-Enhanced Prompt-Evolving for Continual Learning
• Progressive Homeostatic and Plastic Prompt Tuning for Audio-Visual Multi-Task Incremental Learning
• Graph-Based Continual Learning
• ViRN: Variational Inference and Distribution Trilateration for Long-Tailed Continual Representation Learning

Related

• A Novel Approach to Lifelong Learning: The Plastic Support Structure
• Lightweight Learner for Shared Knowledge Lifelong Learning
• Lifelong Learning Starting From Zero
• Lifelong Twin Generative Adversarial Networks
• Task-Aware Multi-Expert Architecture For Lifelong Deep Learning