GaLore 2: Large-Scale LLM Pre-Training by Gradient Low-Rank Projection

2025/04/29 by DiJia Su, Su, DiJia, Andrew Gu +7 · 3 citations
Computer Science · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2504.20437

openalex publication_date 2025/04/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large language models (LLMs) have revolutionized natural language understanding and generation but face significant memory bottlenecks during training. GaLore, Gradient Low-Rank Projection, addresses this issue by leveraging the inherent low-rank structure of weight gradients, enabling substantial memory savings without sacrificing performance. Recent works further extend GaLore from various aspects, including low-bit quantization and higher-order tensor structures. However, there are several remaining challenges for GaLore, such as the computational overhead of SVD for subspace updates and the integration with state-of-the-art training parallelization strategies (e.g., FSDP). In this paper, we present GaLore 2, an efficient and scalable GaLore framework that addresses these challenges and incorporates recent advancements. In addition, we demonstrate the scalability of GaLore 2 by pre-training Llama 7B from scratch using up to 500 billion training tokens, highlighting its potential impact on real LLM pre-training scenarios.

Cited by

• Reparameterized LLM Training via Orthogonal Equivalence Transformation
• Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditioner
• Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarking

Related

• GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
• GaLore+: Boosting Low-Rank Adaptation for LLMs with Cross-Head Projection
• Gradient Weight-normalized Low-rank Projection for Efficient LLM Training
• Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients
• Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarking