DP-FedLoRA: Privacy-Enhanced Federated Fine-Tuning for On-Device Large Language Models

2025/09/11 by Xu, Honghui, Shrestha, Shiva, Chen, Wei +2
#Artificial Intelligence (cs.AI) #Cryptography and Security (cs.CR) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2509.09097

Abstract

As on-device large language model (LLM) systems become increasingly prevalent, federated fine-tuning enables advanced language understanding and generation directly on edge devices; however, it also involves processing sensitive, user-specific data, raising significant privacy concerns within the federated learning framework. To address these challenges, we propose DP-FedLoRA, a privacy-enhanced federated fine-tuning framework that integrates LoRA-based adaptation with differential privacy in a communication-efficient setting. Each client locally clips and perturbs its LoRA matrices using Gaussian noise to satisfy (ε, δ)-differential privacy. We further provide a theoretical analysis demonstrating the unbiased nature of the updates and deriving bounds on the variance introduced by noise, offering practical guidance for privacy-budget calibration. Experimental results across mainstream benchmarks show that DP-FedLoRA delivers competitive performance while offering strong privacy guarantees, paving the way for scalable and privacy-preserving LLM deployment in on-device environments.

Citations

• A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learning
• Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directions
• Towards Label-Only Membership Inference Attack against Pre-trained Large Language Models
• BIG-Bench Extra Hard
• Membership Inference Attacks Against Vision-Language Models
• Comparative Analysis of Efficient Adapter-Based Fine-Tuning of State-of-the-Art Transformer Models
• Federated Fine-Tuning of LLMs: Framework Comparison and Research Directions
• AutoRank: MCDA Based Rank Personalization for LoRA-Enabled Distributed Learning
• IterIS: Iterative Inference-Solving Alignment for LoRA Merging
• A Review on Edge Large Language Models: Design, Execution, and Applications
• FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations
• On-Device Language Models: A Comprehensive Review
• RBLA: Rank-Based-LoRA-Aggregation for Fine-tuning Heterogeneous Models in FLaaS
• LoRA-GA: Low-Rank Adaptation with Gradient Approximation
• The Impact of Initialization on LoRA Finetuning Dynamics
• MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
• LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report
• Not All Federated Learning Algorithms Are Created Equal: A Performance Evaluation Study
• Higher Layers Need More LoRA Experts
• Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment
• On the Role of Server Momentum in Federated Learning
• Accelerating Fair Federated Learning: Adaptive Federated Adam
• BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
• LoRA: Low-Rank Adaptation of Large Language Models
• On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation
• Decentralized Federated Averaging
• Federated Optimization in Heterogeneous Networks
• Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmark

Related

• Differentially Private Federated Low Rank Adaptation Beyond Fixed-Matrix
• FedShield-LLM: A Secure and Scalable Federated Fine-Tuned Large Language Model
• Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learning
• Promoting Data and Model Privacy in Federated Learning through Quantized LoRA
• DP-DyLoRA: Fine-Tuning Transformer-Based Models On-Device under Differentially Private Federated Learning using Dynamic Low-Rank Adaptation