The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting -- An Analytical Model

2024/01/23 by Daniel Goldfarb, Itay Evron, Goldfarb, Daniel +7 · 7 citations
Computer Science · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2401.12617

openalex publication_date 2024/01/23 · openalex created_date 2024/01/26 · openalex updated_date 2026/08/01

Abstract

In continual learning, catastrophic forgetting is affected by multiple aspects of the tasks. Previous works have analyzed separately how forgetting is affected by either task similarity or overparameterization. In contrast, our paper examines how task similarity and overparameterization jointly affect forgetting in an analyzable model. Specifically, we focus on two-task continual linear regression, where the second task is a random orthogonal transformation of an arbitrary first task (an abstraction of random permutation tasks). We derive an exact analytical expression for the expected forgetting - and uncover a nuanced pattern. In highly overparameterized models, intermediate task similarity causes the most forgetting. However, near the interpolation threshold, forgetting decreases monotonically with the expected task similarity. We validate our findings with linear regression on synthetic data, and with neural networks on established permutation task benchmarks.

Cited by

• On Understanding of the Dynamics of Model Capacity in Continual Learning
• Fast Last-Iterate Convergence of SGD in the Smooth Interpolation Regime
• The Importance of Being Lazy: Scaling Limits of Continual Learning
• Optimal Rates in Continual Linear Regression via Increasing Regularization
• Unlocking the Power of Rehearsal in Continual Learning: A Theoretical Perspective
• From Continual Learning to SGD and Back: Better Rates for Continual Linear Models
• Memory-Statistics Tradeoff in Continual Learning with Structural Regularization

Related

• Analysis of Overparameterization in Continual Learning under a Linear Model
• Understanding Forgetting in Continual Learning with Linear Regression
• Analysis of Catastrophic Forgetting for Random Orthogonal Transformation Tasks in the Overparameterized Regime
• Theory on Forgetting and Generalization of Continual Learning
• Toward Understanding Catastrophic Forgetting in Continual Learning