On Robust Incremental Learning over Many Multilingual Steps

2022/10/25 by Karan Praharaj, Praharaj, Karan, Irina Matveeva +1
Computer Science · #Computation and Language (cs.CL) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Speech Recognition and Synthesis

paper · pdf · doi:10.48550/arxiv.2210.14307

openalex publication_date 2022/10/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent work in incremental learning has introduced diverse approaches to tackle catastrophic forgetting from data augmentation to optimized training regimes. However, most of them focus on very few training steps. We propose a method for robust incremental learning over dozens of fine-tuning steps using data from a variety of languages. We show that a combination of data-augmentation and an optimized training regime allows us to continue improving the model even for as many as fifty training steps. Crucially, our augmentation strategy does not require retaining access to previous training data and is suitable in scenarios with privacy constraints.

Related

• Overcoming Catastrophic Forgetting in Massively Multilingual Continual Learning
• Unlearning at Scale: Implementing the Right to be Forgotten in Large Language Models
• A Strategy for an Uncompromising Incremental Learner
• Learn or Recall? Revisiting Incremental Learning with Pre-trained Language Models
• Reminding the Incremental Language Model via Data-Free Self-Distillation