Kron-LoRA: Hybrid Kronecker-LoRA Adapters for Scalable, Sustainable Fine-tuning

2025/08/04 by Shen, Yixin · 1 citation
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2508.01961

Abstract

Fine-tuning massive pre-trained language models across many tasks demands adapters that are both parameter-efficient and expressive. We introduce Kron-LoRA, a hybrid adapter that combines Kronecker-structured factorization with low-rank LoRA compression-an integration that, to our knowledge, has not been explored in parameter-efficient fine-tuning or in matrix approximation literature. Kron-LoRA achieves up to 4× fewer parameters than standard LoRA while retaining similar expressivity. Experiments on DistilBERT, Mistral-7B, LLaMA-2-7B, and LLaMA-3-8B across eight benchmarks show that Kron-LoRA matches or exceeds LoRA baselines with modest memory savings and only a 5-8% speed overhead. In sequential fine-tuning, it also delivers competitive cross-task transfer despite using only one-quarter of the adapter parameters. Kron-LoRA thus offers a scalable, sustainable solution for multi-task adaptation of large language models.

Citations

• The Llama 3 Herd of Models
• MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning
• DoRA: Weight-Decomposed Low-Rank Adaptation
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• QLoRA: Efficient Finetuning of Quantized LLMs
• LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models
• AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
• DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation
• Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting
• PIQA: Reasoning about Physical Commonsense in Natural Language
• DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
• HellaSwag: Can a Machine Really Finish Your Sentence?
• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
• Parameter-Efficient Transfer Learning for NLP
• Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Prefix-Tuning: Optimizing Continuous Prompts for Generation

Cited by

• On the Convergence Rate of LoRA Gradient Descent

Related

• LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters
• CE-LoRA: Computation-Efficient LoRA Fine-Tuning for Language Models
• VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks
• LoRA-Mini : Adaptation Matrices Decomposition and Selective Training
• Uni-LoRA: One Vector is All You Need