Optimal Training-Time Scaling in Gradual Adaptation

2026/08/05 by Zonghuan Xu, Krishna Harish
Computer Science · #cs.LG

paper · pdf

24 pages, 5 figures

arxiv created 2026/08/06 · arxiv updated 2026/08/07

Abstract

In gradual adaptation, how should the training time on each task change as the number of intermediate tasks increases? We study this question for overparameterized linear regression tasks that change smoothly and share a zero-loss solution. With N tasks and training time s_N on each, the final learning progress converges to a continuum curve when Ns_N→τ. The limiting progress is Θ(τ) for small τ and Θ(τ^-1) for large τ, so both very short and very long training produce little progress. It follows that optimal per-task training times scale as s_N^⋆=Θ(N^-1), equivalently Ns_N^⋆=Θ(1). Experiments on gradually rotated MNIST and a natural Yearbook time shift are consistent with less per-task training as the path is divided more finely.

Citations

• The Importance of Being Lazy: Scaling Limits of Continual Learning
• Optimal Rates in Continual Linear Regression via Increasing Regularization
• From Continual Learning to SGD and Back: Better Rates for Continual Linear Models
• Optimal Protocols for Continual Learning via Statistical Physics and Control Theory
• DPCore: Dynamic Prompt Coreset for Continual Test-Time Adaptation
• A Statistical Theory of Regularization-Based Continual Learning
• Robust Test-Time Adaptation in Dynamic Scenarios
• Dynamic Update-to-Data Ratio: Minimizing World Model Overfitting
• Towards Stable Test-Time Adaptation in Dynamic Wild World
• Test-time Adaptation in the Dynamic World with Compound Domain Knowledge Management
• Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time
• Robust Mean Teacher for Continual and Gradual Test-Time Adaptation
• Gradual Domain Adaptation without Indexed Intermediate Domains
• Challenging Common Assumptions about Catastrophic Forgetting
• Understanding Gradual Domain Adaptation: Improved Analysis, Optimal Path and Beyond
• Efficient Test-Time Model Adaptation without Forgetting
• Continual Test-Time Domain Adaptation
• MEMO: Test Time Robustness via Adaptation and Augmentation
• Understanding the Role of Training Regimes in Continual Learning
• Understanding Self-Training for Gradual Domain Adaptation
• Orthogonal Gradient Descent for Continual Learning
• A continual learning survey: Defying forgetting in classification tasks
• On The Power of Curriculum Learning in Training Deep Networks
• A Continuous-Time View of Early Stopping for Least Squares
• Efficient Lifelong Learning with A-GEM
• Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting
• Continual Lifelong Learning with Neural Networks: A Review
• Maximum Classifier Discrepancy for Unsupervised Domain Adaptation
• CyCADA: Cycle-Consistent Adversarial Domain Adaptation
• Gradient Episodic Memory for Continual Learning
• Automated Curriculum Learning for Neural Networks
• Overcoming catastrophic forgetting in neural networks
• A Century of Portraits: A Visual Historical Record of American High School Yearbooks
• Train faster, generalize better: Stability of stochastic gradient descent
• Domain-Adversarial Training of Neural Networks
• Early stopping and non-parametric regression: An optimal data-dependent stopping rule
• Adiabatic theorems for generators of contracting evolutions
• On Early Stopping in Gradient Descent Learning