Think Locally, Act Globally: Federated Learning with Local and Global Representations

2020/01/06 by Paul Pu Liang, Liang, Paul Pu, Terrance Liu +13 · 16 citations
Computer Science · Social Sciences · #Distributed #FOS: Computer and information sciences #Human Mobility and Location-Based Analysis #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Parallel #Privacy, Security, and Data Protection #Privacy-Preserving Technologies in Data #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2001.01523

openalex publication_date 2020/01/06 · openalex created_date 2022/07/26 · openalex updated_date 2026/07/28

Abstract

Federated learning is a method of training models on private data distributed over multiple devices. To keep device data private, the global model is trained by only communicating parameters and updates which poses scalability challenges for large models. To this end, we propose a new federated learning algorithm that jointly learns compact local representations on each device and a global model across all devices. As a result, the global model can be smaller since it only operates on local representations, reducing the number of communicated parameters. Theoretically, we provide a generalization analysis which shows that a combination of local and global models reduces both variance in the data as well as variance across device distributions. Empirically, we demonstrate that local models enable communication-efficient training while retaining performance. We also evaluate on the task of personalized mood prediction from real-world mobile data where privacy is key. Finally, local models handle heterogeneous data from new devices, and learn fair representations that obfuscate protected attributes such as race, age, and gender.

Cited by

• FedMPDD: Communication-Efficient Federated Learning with Privacy Preservation Attributes via Projected Directional Derivative
• FedRE: A Representation Entanglement Framework for Model-Heterogeneous Federated Learning
• Lightweight Federated Learning in Mobile Edge Computing with Statistical and Device Heterogeneity Awareness
• FedTopo: Relation-Level Topology Sharing for Model-Heterogeneous Federated Learning
• Privacy-Aware Federated nnU-Net for ECG Page Digitization
• Federated Learning via Meta-Variational Dropout
• CO-PFL: Contribution-Oriented Personalized Federated Learning for Heterogeneous Networks
• Machine Olfaction and Embedded AI Are Shaping the New Global Sensing Industry
• FedDEAP: Adaptive Dual-Prompt Tuning for Multi-Domain Federated Learning
• Personalized Collaborative Learning with Affinity-Based Variance Reduction
• Choice Outweighs Effort: Facilitating Complementary Knowledge Fusion in Federated Learning via Re-calibration and Merit-discrimination
• Towards Privacy-Preserving and Heterogeneity-aware Split Federated Learning via Probabilistic Masking
• AI-in-the-Loop: Privacy Preserving Real-Time Scam Detection and Conversational Scambaiting by Leveraging LLMs and Federated Learning
• Dual-Distilled Heterogeneous Federated Learning with Adaptive Margins for Trainable Global Prototypes
• FedUNet: A Lightweight Additive U-Net Module for Federated Learning with Heterogeneous Models
• Generalizable Federated Learning using Client Adaptive Focal Modulation

Related

• FedeCouple: Fine-Grained Balancing of Global-Generalization and Local-Adaptability in Federated Learning
• Evaluating the Communication Efficiency in Federated Learning Algorithms
• Incentive Design for Efficient Federated Learning in Mobile Networks: A Contract Theory Approach
• FedRIR: Rethinking Information Representation in Federated Learning
• Federated Learning of a Mixture of Global and Local Models