2020/04/07 by Germán Kruszewski, Ionut Sorodoc, Ionut-Teodor Sorodoc +5 · 7 citations
Computer Science · #Artificial Intelligence (cs.AI) #Artificial intelligence #Computation and Language (cs.CL) #Computer science #FOS: Computer and information sciences #Intelligent Tutoring Systems and Adaptive Learning #Machine Learning (cs.LG) #Machine Learning and Algorithms #Multimodal Machine Learning Applications #cs.AI #cs.CL #cs.LG
paper · pdf · doi:10.48550/arxiv.2004.03340
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2020/04/07 · arxiv created 2021/02/01 · arxiv updated 2021/02/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Online Continual Learning (OCL) studies learning over a continuous data stream without observing any single example more than once, a setting that is closer to the experience of humans and systems that must learn "on-the-wild". Yet, commonly available benchmarks are far from these real-world conditions, because they explicitly signal different tasks, lack latent similarity structure or assume temporal independence between different examples. Here, we propose a new benchmark for OCL based on language modelling in which input alternates between different languages and domains without any explicit delimitation. Additionally, we propose new metrics to study catastrophic forgetting in this setting and evaluate multiple baseline models based on compositions of experts. Finally, we introduce a simple gating technique that learns the latent similarities between different inputs, improving the performance of a Products of Experts model.