Communication-Efficient and Accurate Approach for Aggregation in Federated Low-Rank Adaptation

2025/09/30 by Minh-Duong Nguyen, Nguyen, Le-Tuan, Nguyen, Minh-Duong +6
Computer Science · Engineering · #68 #Advanced Photonic Communication Systems #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #I.2 #Image and Video Quality Assessment

paper · pdf · doi:10.48550/arxiv.2509.26399

openalex publication_date 2025/09/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

With the rapid emergence of foundation models and the increasing need for fine-tuning across distributed environments, Federated Low-Rank Adaptation (FedLoRA) has recently gained significant attention. Despite enormous potential, current FedLoRA methods face notable challenges due to inexact updates. Existing approaches have attempted to mitigate this issue, but they often introduce a local-global generalization gap and incur substantial communication overhead, limiting their scalability and effectiveness. To address these limitations, we propose Federated Low-Rank Aggregation with Nearly Accurate Estimation (FLoRA-NA). FLoRA-NA leverages the local LoRA matrices on the server to estimate the aggregated matrices A and B, which are then distributed to clients for local updates. This surrogated aggregated matrices minimizes the divergence between ideal ∇ \BarW = ∑^U_u=1B_u A_u and practical updates ∇ W = BA without adding communication cost beyond vanilla FedLoRA. By doing so, FLoRA-NA achieves communication efficiency and bridges the gap between local personalization and global generalization, addressing a key limitation of prior personalized FedLoRA approaches. We conduct extensive evaluations across diverse tasks, including natural language understanding, mathematical reasoning, and code-solving ability using various foundation models. Experimental results consistently demonstrate that FLoRA-NA achieves state-of-the-art global performance while maintaining low communication overhead.

Citations

• Enhancing Gradient Variance and Differential Privacy in Quantum Federated Learning
• Improving Generalization in Heterogeneous Federated Continual Learning via Spatio-Temporal Gradient Matching with Prototypical Coreset
• Qwen3 Technical Report
• LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Models
• Federated Domain Generalization with Data-free On-server Matching Gradient
• Multi-Agent Collaboration Mechanisms: A Survey of LLMs
• Selective Aggregation for Low-Rank Adaptation in Federated Learning
• FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations
• Wicked Oddities: Selectively Poisoning for Effective Clean-Label Backdoor Attacks
• Towards Federated Low-Rank Adaptation of Language Models with Rank Heterogeneity
• FDLoRA: Personalized Federated Learning of Large Language Model via Dual LoRA Tuning
• Dual-Personalizing Adapter for Federated Foundation Models
• Improving LoRA in Privacy-preserving Federated Learning
• Gemma: Open Models Based on Gemini Research and Technology
• SPEAR:Exact Gradient Inversion of Batches in Federated Learning
• Global and Local Prompts Cooperation via Optimal Transport for Federated Learning
• OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement
• DoRA: Weight-Decomposed Low-Rank Adaptation
• Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models
• Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learning
• Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes
• A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA
• Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmark
• Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learning
• pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuning
• From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models
• MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models
• FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning
• LoRA-FA: Memory-efficient Low-rank Adaptation for Large Language Models Fine-tuning
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• QLoRA: Efficient Finetuning of Quantized LLMs
• Towards Building the Federated GPT: Federated Instruction Tuning
• GPT-4 Technical Report
• FedExP: Speeding Up Federated Averaging via Extrapolation
• HCFL: A High Compression Approach for Communication-Efficient Federated Learning in Very Large Scale IoT Networks
• FederatedScope: A Flexible Federated Learning Platform for Heterogeneity
• Training Verifiers to Solve Math Word Problems
• Program Synthesis with Large Language Models
• Evaluating Large Language Models Trained on Code
• LoRA: Low-Rank Adaptation of Large Language Models
• Intrinsic Dimensionality Explains the Effectiveness of Language Model\n Fine-Tuning
• Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization
• Personalized Federated Learning with Moreau Envelopes
• Language Models are Few-Shot Learners
• Advances and Open Problems in Federated Learning
• Advances and Open Problems in Federated Learning
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• Parameter-Efficient Transfer Learning for NLP
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Learning multiple visual domains with residual adapters
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Prefix-Tuning: Optimizing Continuous Prompts for Generation

Related

• FedHL: Federated Learning for Heterogeneous Low-Rank Adaptation via Unbiased Aggregation
• FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Foundation Models
• FLoRIST: Singular Value Thresholding for Efficient and Accurate Federated Fine-Tuning of Large Language Models
• AFLoRA: Adaptive Federated Fine-Tuning of Large Language Models with Resource-Aware Low-Rank Adaption
• FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations