2024/06/14 by Guy Hacohen, Tinne Tuytelaars, Hacohen, Guy +1 · 1 citation
Social Sciences · #Education and Critical Thinking Development #FOS: Computer and information sciences #Higher Education Learning Practices #Machine Learning (cs.LG)
paper · pdf · doi:10.48550/arxiv.2406.09935
openalex publication_date 2024/06/14 · openalex created_date 2024/06/18 · openalex updated_date 2026/07/28
Catastrophic forgetting - the tendency of neural networks to forget previously learned data when learning new information - remains a central challenge in continual learning. In this work, we adopt a behavioral approach, observing a connection between learning speed and forgetting: examples learned more quickly are less prone to forgetting. Focusing on replay-based continual learning, we show that the composition of the replay buffer - specifically, whether it contains quickly or slowly learned examples - has a significant effect on forgetting. Motivated by this insight, we introduce Speed-Based Sampling (SBS), a simple yet general strategy that selects replay examples based on their learning speed. SBS integrates easily into existing buffer-based methods and improves performance across a wide range of competitive continual learning benchmarks, advancing state-of-the-art results. Our findings underscore the value of accounting for the forgetting dynamics when designing continual learning algorithms.