Subspace Optimization for Large Language Models with Convergence Guarantees

2024/10/15 by Yutong He, He, Yutong, Peichao Li +7 · 16 citations
Computer Science · #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Optimization and Control (math.OC) #Speech Recognition and Synthesis #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2410.11289

openalex publication_date 2024/10/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Subspace optimization algorithms, such as GaLore (Zhao et al., 2024), have gained attention for pre-training and fine-tuning large language models (LLMs) due to their memory efficiency. However, their convergence guarantees remain unclear, particularly in stochastic settings. In this paper, we reveal that GaLore does not always converge to the optimal solution and provide an explicit counterexample to support this finding. We further explore the conditions under which GaLore achieves convergence, showing that it does so when either (i) a sufficiently large mini-batch size is used or (ii) the gradient noise is isotropic. More significantly, we introduce GoLore (Gradient random Low-rank projection), a novel variant of GaLore that provably converges in typical stochastic settings, even with standard batch sizes. Our convergence analysis extends naturally to other subspace optimization algorithms. Finally, we empirically validate our theoretical results and thoroughly test the proposed mechanisms. Codes are available at https://github.com/pkumelon/Golore.

Cited by

• On the Convergence Rate of LoRA Gradient Descent
• Parameter-Efficient Subspace Optimization for LLM Fine-Tuning
• An All-Reduce Compatible Top-K Compressor for Communication-Efficient Distributed Learning
• MeCeFO: Enhancing LLM Training Robustness via Fault-Tolerant Optimization
• Unbiased Gradient Low-Rank Projection
• Geometrically Principled Randomized Optimization for Efficient LLM Training
• AdaPM: a Partial Momentum Algorithm for LLM Training
• Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptation
• CR-Net: Scaling Parameter-Efficient Training with Cross-Layer Low-Rank Structure
• From PowerSGD to PowerSGD+: Low-Rank Gradient Compression for Distributed Optimization with Convergence Guarantees
• An Efficient Subspace Algorithm for Federated Learning on Heterogeneous Data
• Greedy Low-Rank Gradient Compression for Distributed Learning with Convergence Guarantees
• Memory-Efficient Differentially Private Training with Gradient Random Projection
• MLorc: Momentum Low-rank Compression for Memory Efficient Large Language Model Adaptation
• Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarking
• AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projections

Related

• A Memory Efficient Randomized Subspace Optimization Method for Training Large Language Models
• Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretraining
• Zeroth-Order Fine-Tuning of LLMs in Random Subspaces
• Geometrically Principled Randomized Optimization for Efficient LLM Training
• Unbiased Gradient Low-Rank Projection