ReLoRA: High-Rank Training Through Low-Rank Updates

2023/07/11 by Vladislav Lialin, Namrata Shivagunde, Lialin, Vladislav +5 · 1 voice · 30 citations
Computer Science · #Advanced Neural Network Applications #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural Networks and Applications #Speech Recognition and Synthesis #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2307.05695

openalex publication_date 2023/07/11 · arxiv published 2023/07/11 · arxiv updated 2023/12/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Despite the dominance and effectiveness of scaling, resulting in large networks with hundreds of billions of parameters, the necessity to train overparameterized models remains poorly understood, while training costs grow exponentially. In this paper, we explore parameter-efficient training techniques as an approach to training large neural networks. We introduce a novel method called ReLoRA, which utilizes low-rank updates to train high-rank networks. We apply ReLoRA to training transformer language models with up to 1.3B parameters and demonstrate comparable performance to regular neural network training. ReLoRA saves up to 5.5Gb of RAM per GPU and improves training speed by 9-40% depending on the model size and hardware setup. Our findings show the potential of parameter-efficient techniques for large-scale pre-training.

Cited by

• On the Convergence Rate of LoRA Gradient Descent
• Bridging Training and Merging Through Momentum-Aware Optimization
• BOOST: BOttleneck-Optimized Scalable Training Framework for Low-Rank Large Language Models
• FOAM: Blocked State Folding for Memory-Efficient LLM Training
• EcoSpa: Efficient Transformer Training with Coupled Sparsity
• Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training
• LoRAQuant: Mixed-Precision Quantization of LoRA to Ultra-Low Bits
• IBNorm: Information-Bottleneck Inspired Normalization for Representation Learning
• MISA: Memory-Efficient LLMs Optimization with Module-wise Importance Sampling
• Calibrating and Rotating: A Unified Framework for Weight Conditioning in PEFT
• Kernelized Sparse Fine-Tuning with Bi-level Parameter Competition for Vision Models
• FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmetic
• MeCeFO: Enhancing LLM Training Robustness via Fault-Tolerant Optimization
• ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuning
• Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation
• ADARL: Adaptive Low-Rank Structures for Robust Policy Learning under Uncertainty
• StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold
• Randomized Gradient Subspaces for Efficient Large Language Model Training
• Accelerating Attention with Basis Decomposition
• AdaPM: a Partial Momentum Algorithm for LLM Training
• HoRA: Cross-Head Low-Rank Adaptation with Joint Hypernetworks
• PreLoRA: Hybrid Pre-training of Vision Transformers with Full Training and Low-Rank Adapters
• Faster Than SVD, Smarter Than SGD: The OPLoRA Alternating Update
• CR-Net: Scaling Parameter-Efficient Training with Cross-Layer Low-Rank Structure
• Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning
• Pico: A Modular Framework for Hypothesis-Driven Small Language Model Research
• Investigating ReLoRA: Effects on the Learning Dynamics of Small Language Models
• FedReFT: Federated Representation Fine-Tuning with All-But-Me Aggregation
• BoRA: Towards More Expressive Low-Rank Adaptation with Block Diversity
• LOST: Low-rank and Sparse Pre-training for Large Language Models

Discussions

• ReLoRA: High-Rank Training Through Low-Rank Updates [hn, 3 points, 0 comments]

Related

• RandLoRA: Full-rank parameter-efficient fine-tuning of large models
• BOOST: BOttleneck-Optimized Scalable Training Framework for Low-Rank Large Language Models
• Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training
• CR-Net: Scaling Parameter-Efficient Training with Cross-Layer Low-Rank Structure
• Training Neural Networks from Scratch with Parallel Low-Rank Adapters