CPR: Classifier-Projection Regularization for Continual Learning

2020/06/12 by Sungmin Cha, Hsiang Hsu, Cha, Sungmin +8 · 32 citations
Computer Science · Mathematics · #Artificial intelligence #Classifier (UML) #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Conditional entropy #Domain Adaptation and Few-Shot Learning #Entropy (arrow of time) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Machine learning #Mathematics #Maxima and minima #Multimodal Machine Learning Applications #Pattern recognition (psychology) #Principle of maximum entropy #Regularization (linguistics) #cs.CV #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2006.07326

published in arXiv (Cornell University) (Cornell University) · ICLR 2021 camera ready version

openalex publication_date 2020/06/12 · arxiv created 2021/04/19 · arxiv updated 2021/04/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/08

Abstract

We propose a general, yet simple patch that can be applied to existing regularization-based continual learning methods called classifier-projection regularization (CPR). Inspired by both recent results on neural networks with wide local minima and information theory, CPR adds an additional regularization term that maximizes the entropy of a classifier's output probability. We demonstrate that this additional term can be interpreted as a projection of the conditional probability given by a classifier's output to the uniform distribution. By applying the Pythagorean theorem for KL divergence, we then prove that this projection may (in theory) improve the performance of continual learning methods. In our extensive experimental results, we apply CPR to several state-of-the-art regularization-based continual learning methods and benchmark performance on popular image recognition datasets. Our results demonstrate that CPR indeed promotes a wide local minima and significantly improves both accuracy and plasticity while simultaneously mitigating the catastrophic forgetting of baseline continual learning methods. The codes and scripts for this work are available at https://github.com/csm9493/CPR_CL.

Cited by

• Dynamic Feedback Engines: Layer-Wise Control for Self-Regulating Continual Learning
• Continual Knowledge Consolidation LORA for Domain Incremental Learning
• Logits Replay + MoClip: Stabilized, Low-Cost Post-Training with Minimal Forgetting
• Source-Free Cross-Domain Continual Learning
• Understanding Catastrophic Interference: On the Identifibility of Latent Representations
• C-Flat++: Towards a More Efficient and Powerful Framework for Continual Learning
• The Bayesian Approach to Continual Learning: An Overview
• Branch, or Layer? Zeroth-Order Optimization for Continual Learning of Vision-Language Models
• Balanced Online Class-Incremental Learning via Dual Classifiers
• SEE: Continual Fine-tuning with Sequential Ensemble of Experts

Related

• Optimization and Generalization of Regularization-Based Continual Learning: a Loss Approximation Viewpoint
• DLCFT: Deep Linear Continual Fine-Tuning for General Incremental Learning
• Regularization Shortcomings for Continual Learning
• Regularizing Neural Networks by Penalizing Confident Output Distributions
• A Closer Look at Rehearsal-Free Continual Learning