Class-Incremental Learning based on Label Generation

2023/06/22 by Shao, Yijia, Guo, Yiduo, Zhao, Dongyan +1
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural and Evolutionary Computing (cs.NE)

paper · doi:10.48550/arxiv.2306.12619

Abstract

Despite the great success of pre-trained language models, it is still a challenge to use these models for continual learning, especially for the class-incremental learning (CIL) setting due to catastrophic forgetting (CF). This paper reports our finding that if we formulate CIL as a continual label generation problem, CF is drastically reduced and the generalizable representations of pre-trained models can be better retained. We thus propose a new CIL method (VAG) that also leverages the sparsity of vocabulary to focus the generation and creates pseudo-replay samples by using label semantics. Experimental results show that VAG outperforms baselines by a large margin.

Related

• ClaRe: Practical Class Incremental Learning By Remembering Previous Class Representations
• Dynamic Prompt Adjustment for Multi-Label Class-Incremental Learning
• Balancing the Causal Effects in Class-Incremental Learning
• Knowledge Restore and Transfer for Multi-label Class-Incremental Learning
• Make Domain Shift a Catastrophic Forgetting Alleviator in Class-Incremental Learning