Towards Symmetric Low-Rank Adapters

2025/03/29 by Tales Panoutsos, Panoutsos, Tales, Rodrygo L. T. Santos +3
Computer Science · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2504.03719

openalex publication_date 2025/03/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we introduce Symmetric Low-Rank Adapters, an optimized variant of LoRA with even fewer weights. This method utilizes Low-Rank Symmetric Weight Matrices to learn downstream tasks more efficiently. Traditional LoRA accumulates fine-tuning weights with the original pre-trained weights via a Singular Value Decomposition (SVD) like approach, i.e., model weights are fine-tuned via updates of the form BA (where B ∈ ℝ^{n× r}, A ∈ ℝ^{r× n}, and r is the rank of the merged weight matrix). In contrast, our approach, named SymLoRA, represents fine-tuning weights as a Spectral Decomposition, i.e., Q diag(Λ) Q^T, where Q ∈ ℝ^{n× r} and Λ∈ ℝ^r. SymLoRA requires approximately half of the finetuning weights. Here, we show that this approach has negligible losses in downstream efficacy.

Related

• A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA
• EDoRA: Efficient Weight-Decomposed Low-Rank Adaptation via Singular Value Decomposition
• LoRA+: Efficient Low Rank Adaptation of Large Models
• LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization
• SARA: Singular-Value Based Adaptive Low-Rank Adaption