LoRA+: Efficient Low Rank Adaptation of Large Models

2024/02/19 by Soufiane Hayou, N. C. Ghosh, Nikhil Ghosh +4 · 3 voices · 67 citations
Computer Science · #Domain Adaptation and Few-Shot Learning #Speech Recognition and Synthesis

paper · pdf · doi:10.48550/arxiv.2402.12354

Abstract

In this paper, we show that Low Rank Adaptation (LoRA) as originally introduced in Hu et al. (2021) leads to suboptimal finetuning of models with large width (embedding dimension). This is due to the fact that adapter matrices A and B in LoRA are updated with the same learning rate. Using scaling arguments for large width networks, we demonstrate that using the same learning rate for A and B does not allow efficient feature learning. We then show that this suboptimality of LoRA can be corrected simply by setting different learning rates for the LoRA adapter matrices A and B with a well-chosen ratio. We call this proposed algorithm LoRA+. In our extensive experiments, LoRA+ improves performance (1-2 % improvements) and finetuning speed (up to ∼ 2X SpeedUp), at the same computational cost as LoRA.

Cited by

• On the Convergence of Stochastic Low-Rank Adaptation
• MoE²-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation
• The Blessing of Dimensionality: How Near-Orthogonality in High-Dimensional Spaces Explains Temporal Portability
• PoLoRA: A Preconditioned Orthogonalized LoRA Optimizer
• DynImmune-BERT: Dynamic Immune Repertoire Modeling with Neural ODE Driven Continuous Transformers
• DARA: Degradation-Aware Low-Rank Residual Adaptation with Original-to-Corrupted Distillation for Corruption-Robust Animal Re-Identification
• MIRAGE: The Illusion of Visual Understanding
• In-Context Probing for Membership Inference in Fine-Tuned Language Models
• The Quest for Winning Tickets in Low-Rank Adapters
• The Intruder Threshold: A Spectral Law for LoRA Fine-Tuning
• MoRAgent: Parameter Efficient Agent Tuning with Mixture-of-Roles
• On the Convergence Rate of LoRA Gradient Descent
• Beyond Memorization: Selective Learning for Copyright-Safe Diffusion Model Training
• qa-FLoRA: Data-free query-adaptive Fusion of LoRAs for LLMs
• Efficient Continual Learning in Neural Machine Translation: A Low-Rank Adaptation Approach
• Parameter-Efficient Augment Plugin for Class-Incremental Learning
• PEFT-Factory: Unified Parameter-Efficient Fine-Tuning of Autoregressive Large Language Models
• Parameter-Efficient Subspace Optimization for LLM Fine-Tuning
• CrossEarth-Gate: Fisher-Guided Adaptive Tuning Engine for Efficient Adaptation of Cross-Domain Remote Sensing Semantic Segmentation
• ABM-LoRA: Activation Boundary Matching for Fast Convergence in Low-Rank Adaptation
• Comparative Analysis of LoRA-Adapted Embedding Models for Clinical Cardiology Text Representation
• NP-LoRA: Null Space Projection Unifies Subject and Style in LoRA Fusion
• Between Gradient and Natural Gradient: A Continuum of LoRA Initializations
• Fed-PELAD: Communication-Efficient Federated Learning for Massive MIMO CSI Feedback with Personalized Encoders and a LoRA-Adapted Shared Decoder
• zFLoRA: Zero-Latency Fused Low-Rank Adapters
• Calibrating and Rotating: A Unified Framework for Weight Conditioning in PEFT
• FLoRA: Fused forward-backward adapters for parameter efficient fine-tuning and reducing inference-time latencies of LLMs
• Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation
• α-LoRA: Effective Fine-Tuning via Base Model Rescaling
• GaLLoP: Gradient-based Sparse Learning on Low-Magnitude Parameters
• Latent Space Factorization in LoRA
• Directional Reasoning Injection for Fine-Tuning MLLMs
• ScaleWeaver: Weaving Efficient Controllable T2I Generation with Multi-Scale Reference Attention
• CTR-LoRA: Curvature-Aware and Trust-Region Guided Low-Rank Adaptation for Large Language Models
• StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold
• Efficient Resource-Constrained Training of Vision Transformers via Subspace Optimization
• AILoRA: Function-Aware Asymmetric Initialization for Low-Rank Adaptation of Large Language Models
• HySim-LLM: Embedding-Weighted Fine-Tuning Bounds and Manifold Denoising for Domain-Adapted LLMs
• SliceFine: The Universal Winning-Slice Hypothesis for Pretrained Networks
• Optimizing Fine-Tuning through Advanced Initialization Strategies for Low-Rank Adaptation
• Syntax-Guided Diffusion Language Models with User-Integrated Personalization
• Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptation
• SynthPert: Enhancing LLM Biological Reasoning via Synthetic Reasoning Traces for Cellular Perturbation Prediction
• StolenLoRA: Exploring LoRA Extraction Attacks via Synthetic Data
• Large language models for material property predictions: elastic constant tensor prediction and materials design
• Enhancing Low-Rank Adaptation with Structured Nonlinear Transformations
• PreLoRA: Hybrid Pre-training of Vision Transformers with Full Training and Low-Rank Adapters
• Parameter-Efficient Multi-Task Learning via Progressive Task-Specific Adaptation
• Tight Sample Complexity for Low-Rank Adaptation: Matching Bounds and Rank Selection
• EndoUFM: Utilizing Foundation Models for Monocular depth estimation of endoscopic images
• HyperAdapt: Simple High-Rank Adaptation
• FURINA: Free from Unmergeable Router via LINear Aggregation of mixed experts
• SparseDoctor: Towards Efficient Chat Doctor with Mixture of Experts Enhanced Large Language Models
• PHLoRA: data-free Post-hoc Low-Rank Adapter extraction from full-rank checkpoint
• HEFT: A Coarse-to-Fine Hierarchy for Enhancing the Efficiency and Accuracy of Language Model Reasoning
• One Model for All Tasks: Leveraging Efficient World Models in Multi-Task Planning
• L1RA: Dynamic Rank Assignment in LoRA Fine-Tuning
• IPA: An Information-Reconstructive Input Projection Framework for Efficient Foundation Model Adaptation
• MEPT: Mixture of Expert Prompt Tuning as a Manifold Mapper
• LFD: Layer Fused Decoding to Exploit External Knowledge in Retrieval-Augmented Generation
• Quantized but Deceptive? A Multi-Dimensional Truthfulness Evaluation of Quantized LLMs
• AdaRing: Towards Ultra-Light Vision-Language Adaptation via Cross-Layer Tensor Ring Decomposition
• Nested-ReFT: Efficient Reinforcement Learning for Large Language Model Fine-Tuning via Off-Policy Rollouts
• BoRA: Towards More Expressive Low-Rank Adaptation with Block Diversity
• Cross-LoRA: A Data-Free LoRA Transfer Framework across Heterogeneous LLMs
• Towards Higher Effective Rank in Parameter-efficient Fine-tuning using Khatri--Rao Product
• LoRA-PAR: A Flexible Dual-System LoRA Partitioning Approach to Efficient LLM Fine-Tuning

Discussions

• LoRA+: Efficient Low Rank Adaptation of Large Models [hn, 181 points, 47 comments]
• There is a theoretical paper showing Adam’s momentum is misaligned with LoRA’s projection space and … [bsky, 9 points, 1 comments]
• LoRA+: Efficient Low Rank Adaptation of Large Models (arxiv.org) Main Link | Discussion [bsky, 0 points, 0 comments]

Related

• The Expressive Power of Low-Rank Adaptation
• A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA
• \kappa-LoRA: Condition Numbers Reveal Which LoRA Matrices Worth Updating
• LoRA-GA: Low-Rank Adaptation with Gradient Approximation
• GeLoRA: Geometric Adaptive Ranks For Efficient LoRA Fine-tuning