Adaptive Federated LoRA in Heterogeneous Wireless Networks with Independent Sampling

2025/05/29 by Yanzhao Hou, Jiaxiang Geng, Hou, Yanzhao +11
Computer Science · Engineering · #Distributed Sensor Networks and Detection Algorithms #Energy Efficient Wireless Sensor Networks #FOS: Computer and information sciences #Indoor and Outdoor Localization Technologies #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2505.23555

openalex publication_date 2025/05/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Federated LoRA has emerged as a promising technique for efficiently fine-tuning large language models (LLMs) on distributed devices by reducing the number of trainable parameters. However, existing approaches often inadequately overlook the theoretical and practical implications of system and data heterogeneity, thereby failing to optimize the overall training efficiency, particularly in terms of wall-clock time. In this paper, we propose an adaptive federated LoRA strategy with independent client sampling to minimize the convergence wall-clock time of federated fine-tuning under both computation and communication heterogeneity. We first derive a new convergence bound for federated LoRA with arbitrary and independent client sampling, notably without requiring the stringent bounded gradient assumption. Then, we introduce an adaptive bandwidth allocation scheme that accounts for heterogeneous client resources and system bandwidth constraints. Based on the derived theory, we formulate and solve a non-convex optimization problem to jointly determine the LoRA sketching ratios and sampling probabilities, aiming to minimize wall-clock convergence time. An efficient and low-complexity algorithm is developed to approximate the solution. Finally, extensive experiments demonstrate that our approach significantly reduces wall-clock training time compared to state-of-the-art methods across various models and datasets.

Citations

• Ten Challenging Problems in Federated Foundation Models
• Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clients
• Hierarchical Federated Learning with Multi-Timescale Gradient Correction
• FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations
• Demo: FedCampus: A Real-world Privacy-preserving Mobile Application for Smart Campus via Federated Learning & Analytics
• Towards Federated Low-Rank Adaptation of Language Models with Rank Heterogeneity
• Improving LoRA in Privacy-preserving Federated Learning
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
• Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resources
• OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learning
• Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models
• Towards Building the Federated GPT: Federated Instruction Tuning
• GPT-4 Technical Report
• Communication-Efficient Device Scheduling for Federated Learning Using Stochastic Optimization
• Tackling System and Statistical Heterogeneity for Federated Learning with Adaptive Client Sampling
• Cost-Effective Federated Learning in Mobile Edge Networks
• LoRA: Low-Rank Adaptation of Large Language Models
• The Power of Scale for Parameter-Efficient Prompt Tuning
• Cost-Effective Federated Learning Design
• Federated Learning under Importance Sampling
• Optimal Client Sampling for Federated Learning
• On the Convergence of Local Descent Methods in Federated Learning
• On the Convergence of FedAvg on Non-IID Data
• How to Fine-Tune BERT for Text Classification?
• Federated Sketching LoRA: A Flexible Framework for Heterogeneous Collaborative Fine-Tuning of LLMs
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Prefix-Tuning: Optimizing Continuous Prompts for Generation

Related

• Adaptive Federated Learning in Heterogeneous Wireless Networks with Independent Sampling
• FedsLLM: Federated Split Learning for Large Language Models over Communication Networks
• Communication-Efficient Wireless Federated Fine-Tuning for Large-Scale AI Models
• Adaptive Parameter-Efficient Federated Fine-Tuning on Heterogeneous Devices
• Adaptive Heterogeneous Client Sampling for Federated Learning over Wireless Networks