Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuning

2025/06/05 by Arian Raje, Raje, Arian, Baris Askin +5 · 1 citation
Computer Science · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #IoT and Edge/Fog Computing #Machine Learning (cs.LG) #Privacy-Preserving Technologies in Data

paper · pdf · doi:10.48550/arxiv.2506.05568

openalex publication_date 2025/06/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large language models (LLMs) have not yet effectively leveraged the vast amounts of edge-device data, and federated learning (FL) offers a promising paradigm to collaboratively fine-tune LLMs without transferring private edge data to the cloud. To operate within the computation and communication constraints of edge devices, recent literature on federated fine-tuning of LLMs proposes the use of low-rank adaptation (LoRA) and similar parameter-efficient methods. However, LoRA-based methods suffer from accuracy degradation in FL settings, primarily because of data and computational heterogeneity across clients. We propose Ravan, an adaptive multi-head LoRA method that balances parameter efficiency and model expressivity by reparameterizing the weight updates as the sum of multiple LoRA heads s_iB_iH_iA_i in which only the core matrices H_i and their lightweight scaling factors s_i are trained. These trainable scaling factors let the optimization focus on the most useful heads, recovering a higher-rank approximation of the full update without increasing the number of communicated parameters since clients upload s_iH_i directly. Experiments on vision and language benchmarks show that Ravan improves test accuracy by 2-8% over prior parameter-efficient baselines, making it a robust and scalable solution for federated fine-tuning of LLMs.

Citations

• Communication-Efficient and Personalized Federated Foundation Model Fine-Tuning via Tri-Matrix Adaptation
• Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning
• RandLoRA: Full-rank parameter-efficient fine-tuning of large models
• Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuning
• LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinement
• Selective Aggregation for Low-Rank Adaptation in Federated Learning
• Federated Large Language Models: Current Progress and Future Directions
• FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations
• Robust Federated Finetuning of Foundation Models via Alternating Minimization of LoRA
• LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters
• HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning
• FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decomposition
• Mixture of LoRA Experts
• Improving LoRA in Privacy-preserving Federated Learning
• Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resources
• DoRA: Weight-Decomposed Low-Rank Adaptation
• Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models
• VeRA: Vector-based Random Matrix Adaptation
• LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models
• SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models
• QLoRA: Efficient Finetuning of Quantized LLMs
• Towards Building the Federated GPT: Federated Instruction Tuning
• FedExP: Speeding Up Federated Averaging via Extrapolation
• When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methods
• FedPrompt: Communication-Efficient and Privacy Preserving Prompt Tuning in Federated Learning
• FedVARP: Tackling the Variance Due to Partial Client Participation in Federated Learning
• Federated Learning Based on Dynamic Regularization
• FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learning
• BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
• LoRA: Low-Rank Adaptation of Large Language Models
• Federated Learning with Buffered Asynchronous Aggregation
• Model-Contrastive Federated Learning
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients
• TiFL: A Tier-based Federated Learning System
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension
• Parameter-Efficient Transfer Learning for NLP
• Federated Optimization in Heterogeneous Networks
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAs

Related

• FLoRIST: Singular Value Thresholding for Efficient and Accurate Federated Fine-Tuning of Large Language Models
• Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clients
• Resource-Efficient Federated Fine-Tuning Large Language Models for Heterogeneous Data
• Exploring Gradient Subspaces: Addressing and Overcoming LoRA's Limitations in Federated Fine-Tuning of Large Language Models
• Adaptive Rank Allocation for Federated Parameter-Efficient Fine-Tuning of Language Models