LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits

2025/02/12 by Zhou, Zikai, Zhang, Qizheng, Kumbong, Hermann +1 · 1 citation
#Computation and Language (cs.CL) #FOS: Computer and information sciences #Hardware Architecture (cs.AR) #Machine Learning (cs.LG) #Performance (cs.PF)

paper · doi:10.48550/arxiv.2502.08141

Abstract

Fine-tuning large language models (LLMs) is increasingly costly as models scale to hundreds of billions of parameters, and even parameter-efficient fine-tuning (PEFT) methods like LoRA remain resource-intensive. We introduce LowRA, the first framework to enable LoRA fine-tuning below 2 bits per parameter with minimal performance loss. LowRA optimizes fine-grained quantization - mapping, threshold selection, and precision assignment - while leveraging efficient CUDA kernels for scalable deployment. Extensive evaluations across 4 LLMs and 4 datasets show that LowRA achieves a superior performance-precision trade-off above 2 bits and remains accurate down to 1.15 bits, reducing memory usage by up to 50%. Our results highlight the potential of ultra-low-bit LoRA fine-tuning for resource-constrained environments.

Cited by

• On-the-Fly Adaptation to Quantization: Configuration-Aware LoRA for Efficient Fine-Tuning of Quantized LLMs

Related

• LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report
• LoRA Fine-Tuning Without GPUs: A CPU-Efficient Meta-Generation Framework for LLMs
• PLoRA: Efficient LoRA Hyperparameter Tuning for Large Models
• LoRAFusion: Efficient LoRA Fine-Tuning for LLMs
• LoRAQuant: Mixed-Precision Quantization of LoRA to Ultra-Low Bits