BoRA: Towards More Expressive Low-Rank Adaptation with Block Diversity

2025/08/09 by Shiwei Li, Xiandi Luo, Li, Shiwei +14 · 2 citations
Computer Science · Mathematics · Engineering · #Stochastic Gradient Optimization Techniques #Tensor decomposition and applications #Sparse and Compressive Sensing Techniques

paper · pdf · doi:10.48550/arxiv.2508.06953

Abstract

Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning (PEFT) method widely used in large language models (LLMs). It approximates the update of a pretrained weight matrix W∈ℝ^{m× n} by the product of two low-rank matrices, BA, where A ∈ℝ^{r× n} and B∈ℝ^{m× r} (r≪min\m,n\). Increasing the dimension r can raise the rank of LoRA weights (i.e., BA), which typically improves fine-tuning performance but also significantly increases the number of trainable parameters. In this paper, we propose Block Diversified Low-Rank Adaptation (BoRA), which improves the rank of LoRA weights with a small number of additional parameters. Specifically, BoRA treats the product BA as a block matrix multiplication, where A and B are partitioned into b blocks along the columns and rows, respectively (i.e., A=[A₁,…,A_b] and B=[B₁,…,B_b]^\top). Consequently, the product BA becomes the concatenation of the block products B_iA_j for i,j∈[b]. To enhance the diversity of different block products, BoRA introduces a unique diagonal matrix Σ_i,j ∈ ℝ^{r× r} for each block multiplication, resulting in B_i Σ_i,j A_j. By leveraging these block-wise diagonal matrices, BoRA increases the rank of LoRA weights by a factor of b while only requiring b²r additional parameters. Extensive experiments across multiple datasets and models demonstrate the superiority of BoRA, and ablation studies further validate its scalability.

Citations

• Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics
• The Panaceas for Improving Low-Rank Decomposition in Communication-Efficient Federated Learning
• Qwen2.5 Technical Report
• The Llama 3 Herd of Models
• LoRA-GA: Low-Rank Adaptation with Gradient Approximation
• VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks
• HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning
• PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models
• MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Models
• LoRA+: Efficient Low Rank Adaptation of Large Models
• DoRA: Weight-Decomposed Low-Rank Adaptation
• Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment
• A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA
• The Expressive Power of Low-Rank Adaptation
• ReLoRA: High-Rank Training Through Low-Rank Updates
• Residual Prompt Tuning: Improving Prompt Tuning with Residual Reparameterization
• LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models
• GPT-4 Technical Report
• KronA: Parameter Efficient Tuning with Kronecker Adapter
• AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning
• Towards a Unified View of Parameter-Efficient Transfer Learning
• The Power of Scale for Parameter-Efficient Prompt Tuning
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• Parameter-Efficient Transfer Learning for NLP
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Deep Residual Learning for Image Recognition
• AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Prefix-Tuning: Optimizing Continuous Prompts for Generation

Cited by

• Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation
• Quantifying Distributional Invariance in Causal Subgraph for IRM-Free Graph Generalization
• Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics
• The Panaceas for Improving Low-Rank Decomposition in Communication-Efficient Federated Learning
• FlowCut: Rethinking Redundancy via Information Flow for Efficient Vision-Language Models

Related

• BoRA: Bi-dimensional Weight-Decomposed Low-Rank Adaptation
• Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matrices
• SBoRA: Low-Rank Adaptation with Regional Weight Updates
• Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation
• Less is More: Resource-Efficient Low-Rank Adaptation