Recasting Continual Learning as Sequence Modeling

2023/10/18 by Soochan Lee, Lee, Soochan, Jaehyeon Son +3 · 2 citations
Computer Science · #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Human Pose and Action Recognition #Machine Learning (cs.LG) #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2310.11952

openalex publication_date 2023/10/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

In this work, we aim to establish a strong connection between two significant bodies of machine learning research: continual learning and sequence modeling. That is, we propose to formulate continual learning as a sequence modeling problem, allowing advanced sequence models to be utilized for continual learning. Under this formulation, the continual learning process becomes the forward pass of a sequence model. By adopting the meta-continual learning (MCL) framework, we can train the sequence model at the meta-level, on multiple continual learning episodes. As a specific example of our new formulation, we demonstrate the application of Transformers and their efficient variants as MCL methods. Our experiments on seven benchmarks, covering both classification and regression, show that sequence models can be an attractive solution for general MCL.

Cited by

• Meta-Learning Transformers to Improve In-Context Generalization
• Capturing User Interests from Data Streams for Continual Sequential Recommendation

Related

• Learning Mamba as a Continual Learner: Meta-learning Selective State Space Models for Efficient Continual Learning
• Meta Continual Learning via Dynamic Programming
• Learning Invariant Representation for Continual Learning
• A Meta-Learned Neuron model for Continual Learning
• Meta-Learning Representations for Continual Learning