One Rank at a Time: Cascading Error Dynamics in Sequential Learning

2025/05/28 by Mahtab Alizadeh Vandchali, Vandchali, Mahtab Alizadeh, Fangshuo +4 · 1 citation
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Optimization and Control (math.OC) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2505.22602

openalex publication_date 2025/05/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Sequential learning -- where complex tasks are broken down into simpler, hierarchical components -- has emerged as a paradigm in AI. This paper views sequential learning through the lens of low-rank linear regression, focusing specifically on how errors propagate when learning rank-1 subspaces sequentially. We present an analysis framework that decomposes the learning process into a series of rank-1 estimation problems, where each subsequent estimation depends on the accuracy of previous steps. Our contribution is a characterization of the error propagation in this sequential process, establishing bounds on how errors -- e.g., due to limited computational budgets and finite precision -- affect the overall model accuracy. We prove that these errors compound in predictable ways, with implications for both algorithmic design and stability guarantees.

Citations

• Sequential Learning in the Dense Associative Memory
• Learning in Order! A Sequential Strategy to Learn Invariant Features for Multimodal Sentiment Analysis
• Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferences
• Mixture-of-Subspaces in Low-Rank Adaptation
• Towards Modular LLMs by Building and Reusing a Library of LoRAs
• Training Neural Networks from Scratch with Parallel Low-Rank Adapters
• Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learning
• Continual Learning with Low Rank Adaptation
• Efficient Compression of Overparameterized Deep Models through Low-Dimensional Learning Dynamics
• Orthogonal Subspace Learning for Language Model Continual Learning
• Joint Prompt Optimization of Stacked LLMs using Variational Inference
• The Ideal Continual Learner: An Agent That Never Forgets
• AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
• Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensing
• Multi-Head Adapter Routing for Cross-Task Generalization
• Understanding The Robustness in Vision Transformers
• Fast quantum state reconstruction via accelerated non-convex programming
• EigenGame Unloaded: When playing games is better than optimizing
• EigenGame: PCA as a Nash Equilibrium
• Multi-Task Learning with Deep Neural Networks: A Survey
• Understanding the Difficulty of Training Transformers
• Rethinking Bias-Variance Trade-off for Generalization of Neural Networks
• Orthogonal Gradient Descent for Continual Learning
• Implicit Regularization in Deep Matrix Factorization
• Streaming PCA and Subspace Tracking: The Missing Data Case
• Understanding Batch Normalization
• Robustness via Deep Low-Rank Representations
• IHT dies hard: Provable accelerated Iterative Hard Thresholding
• Learning to Compose Skills
• Provable quantum state tomography via non-convex methods
• Accelerated Stochastic Power Iteration
• An Overview of Multi-Task Learning in Deep Neural Networks
• No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified\n Geometric Analysis
• Understanding deep learning requires rethinking generalization
• Non-square matrix sensing without spurious local minima via the Burer-Monteiro approach
• Learning to learn by gradient descent by gradient descent
• Global Optimality of Local Search for Low Rank Matrix Recovery
• Dropping Convexity for Faster Semi-definite Optimization
• Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guarantees
• Low-rank Solutions of Linear Matrix Equations via Procrustes Flow
• A Convergent Gradient Descent Algorithm for Rank Minimization and Semidefinite Programming from Random Linear Measurements
• Provable Inductive Matrix Completion
• Matrix Recipes for Hard Thresholding Methods
• Guaranteed Rank Minimization via Singular Value Projection
• Matrix Factorization Techniques for Recommender Systems
• ADMiRA: Atomic Decomposition for Minimum Rank Approximation
• Guaranteed Minimum Rank Approximation from Linear Observations by Nuclear Norm Minimization with an Ellipsoidal Constraint
• Exact Matrix Completion via Convex Optimization
• Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization
• A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization
• The role of deliberate practice in the acquisition of expert performance.
• Perturbation bounds in connection with singular value decomposition
• Analysis of a complex of statistical variables into principal components.

Cited by

• Improving Progressive Generation with Decomposable Flow Matching

Related

• Analysis of Overparameterization in Continual Learning under a Linear Model
• Toward Understanding Catastrophic Forgetting in Continual Learning
• Online Learning via Sequential Complexities
• Convergence and Implicit Bias of Gradient Descent on Continual Linear Classification
• Understanding Forgetting in Continual Learning with Linear Regression