2025/04/06 by Itay Evron, Evron, Itay, Ran Levinstein +11 · 3 citations
Computer Science · Engineering · Environmental Science · #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Groundwater flow and contamination studies #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reservoir Engineering and Simulation Methods
paper · pdf · doi:10.48550/arxiv.2504.04579
openalex publication_date 2025/04/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
We study the common continual learning setup where an overparameterized model is sequentially fitted to a set of jointly realizable tasks. We analyze forgetting, defined as the loss on previously seen tasks, after k iterations. For continual linear models, we prove that fitting a task is equivalent to a single stochastic gradient descent (SGD) step on a modified objective. We develop novel last-iterate SGD upper bounds in the realizable least squares setup and leverage them to derive new results for continual learning. Focusing on random orderings over T tasks, we establish universal forgetting rates, whereas existing rates depend on problem dimensionality or complexity and become prohibitive in highly overparameterized regimes. In continual regression with replacement, we improve the best existing rate from O((d-r)/k) to O(min(1/√[4]k, √(d-r)/k, √Tr/k)), where d is the dimensionality and r the average task rank. Furthermore, we establish the first rate for random task orderings without replacement. The resulting rate O(min(1/√[4]T, (d-r)/T)) shows that randomization alone, without task repetition, prevents catastrophic forgetting in sufficiently long task sequences. Finally, we prove a matching O(1/√[4]k) forgetting rate for continual linear classification on separable data. Our universal rates extend to broader methods, such as block Kaczmarz and POCS, illuminating their loss convergence under i.i.d. and single-pass orderings.