2022/10/07 by Andrew Silva, Silva, Andrew, Pradyumna Tambwekar +3
Computer Science · Social Sciences · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Privacy, Security, and Data Protection #Privacy-Preserving Technologies in Data #Recommender Systems and Techniques
paper · pdf · doi:10.48550/arxiv.2210.03766
openalex publication_date 2022/10/07 · openalex created_date 2022/10/12 · openalex updated_date 2026/07/28
Federated learning is a training paradigm that learns from multiple distributed users without aggregating data on a centralized server. Such a paradigm promises the ability to deploy machine-learning at-scale to a diverse population of end-users without first collecting a large, labeled dataset for all possible tasks. As federated learning typically averages learning updates across a decentralized population, there is a growing need for personalization of federated learning systems (i.e conversational agents must be able to personalize to a specific user's preferences). In this work, we propose a new direction for personalization research within federated learning, leveraging both personal embeddings and shared context embeddings. We also present an approach to predict these ``preference'' embeddings, enabling personalization without backpropagation. Compared to state-of-the-art personalization baselines, our approach achieves a 50% improvement in test-time perplexity using 0.001% of the memory required by baseline approaches, and achieving greater sample- and compute-efficiency.