Federated Learning Meets Natural Language Processing: A Survey

2021/07/27 by Ming Liu, Liu, Ming, Stella Ho +9 · 4 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Cryptography and Data Security #Distributed #FOS: Computer and information sciences #Parallel #Privacy-Preserving Technologies in Data #Stochastic Gradient Optimization Techniques #and Cluster Computing (cs.DC) #cs.AI #cs.CL #cs.DC

paper · pdf · doi:10.48550/arxiv.2107.12603

19 pages

arxiv created 2021/07/27 · openalex publication_date 2021/07/27 · arxiv updated 2021/07/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Federated Learning aims to learn machine learning models from multiple decentralized edge devices (e.g. mobiles) or servers without sacrificing local data privacy. Recent Natural Language Processing techniques rely on deep learning and large pre-trained language models. However, both big deep neural and language models are trained with huge amounts of data which often lies on the server side. Since text data is widely originated from end users, in this work, we look into recent NLP models and techniques which use federated learning as the learning framework. Our survey discusses major challenges in federated natural language processing, including the algorithm challenges, system challenges as well as the privacy issues. We also provide a critical review of the existing Federated NLP evaluation methods and tools. Finally, we highlight the current research gaps and future directions.

Citations

• Distilling the Knowledge in a Neural Network
• Long Short-Term Memory
• Factorization tricks for LSTM networks
• Hello Edge: Keyword Spotting on Microcontrollers
• SecureBoost: A Lossless Federated Learning Framework
• One-Shot Federated Learning
• Advances and Open Problems in Federated Learning
• Personalized Federated Learning: A Meta-Learning Approach
• FedNER: Privacy-preserving Medical Named Entity Recognition with Federated Learning
• Federated Learning With Differential Privacy: Algorithms and Performance Analysis
• Ensemble Distillation for Robust Model Fusion in Federated Learning
• Communication-Efficient Federated Distillation
• Robust Federated Learning with Attack-Adaptive Aggregation

Cited by

• Adaptive Federated Distillation for Multi-Domain Non-IID Textual Data
• Efficient Federated Learning with Timely Update Dissemination
• Federated Learning-Enabled Hybrid Language Models for Communication-Efficient Token Transmission
• Assortment of Attention Heads: Accelerating Federated PEFT with Head Pruning and Strategic Client Selection

Related

• FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Tasks
• SEMFED: Semantic-Aware Resource-Efficient Federated Learning for Heterogeneous NLP Tasks
• FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning
• Collaborating Heterogeneous Natural Language Processing Tasks via Federated Learning
• Federated Learning: A Cutting-Edge Survey of the Latest Advancements and Applications