FATE-LLM: A Industrial Grade Federated Learning Framework for Large Language Models

2023/10/16 by Tao Fan, Fan, Tao, Yan Kang +11 · 5 citations
Computer Science · Medicine · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Healthcare and Education #FOS: Computer and information sciences #Machine Learning (cs.LG) #Privacy-Preserving Technologies in Data #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2310.10049

openalex publication_date 2023/10/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs), such as ChatGPT, LLaMA, GLM, and PaLM, have exhibited remarkable performances across various tasks in recent years. However, LLMs face two main challenges in real-world applications. One challenge is that training LLMs consumes vast computing resources, preventing LLMs from being adopted by small and medium-sized enterprises with limited computing resources. Another is that training LLM requires a large amount of high-quality data, which are often scattered among enterprises. To address these challenges, we propose FATE-LLM, an industrial-grade federated learning framework for large language models. FATE-LLM (1) facilitates federated learning for large language models (coined FedLLM); (2) promotes efficient training of FedLLM using parameter-efficient fine-tuning methods; (3) protects the intellectual property of LLMs; (4) preserves data privacy during training and inference through privacy-preserving mechanisms. We release the code of FATE-LLM at https://github.com/FederatedAI/FATE-LLM to facilitate the research of FedLLM and enable a broad range of industrial applications.

Cited by

• Trustworthy and Controllable Professional Knowledge Utilization in Large Language Models with TEE-GPU Execution
• Bringing Federated Learning to Space
• LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systems
• A Comprehensive Data-centric Overview of Federated Graph Learning
• Advancing Financial Engineering with Foundation Models: Progress, Applications, and Challenges

Related

• A Survey on Federated Fine-tuning of Large Language Models
• OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learning
• FedCoT: Federated Chain-of-Thought Distillation for Large Language Models
• FedPT: Federated Proxy-Tuning of Large Language Models on Resource-Constrained Edge Devices
• FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models