Private federated discovery of out-of-vocabulary words for Gboard

2024/04/17 by Sun, Ziteng, Kairouz, Peter, Sun, Haicheng +2 · 3 citations
#Data Structures and Algorithms (cs.DS) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2404.11607

Abstract

The vocabulary of language models in Gboard, Google's keyboard application, plays a crucial role for improving user experience. One way to improve the vocabulary is to discover frequently typed out-of-vocabulary (OOV) words on user devices. This task requires strong privacy protection due to the sensitive nature of user input data. In this report, we present a private OOV discovery algorithm for Gboard, which builds on recent advances in private federated analytics. The system offers local differential privacy (LDP) guarantees for user contributed words. With anonymous aggregation, the final released result would satisfy central differential privacy guarantees with ε = 0.315, δ= 10^-10 for OOV discovery in en-US (English in United States).

Cited by

• Private Frequency Estimation Via Residue Number Systems
• Beyond Epsilon: A Principled QIF Framework for Local Differential Privacy
• Toward provably private analytics and insights into GenAI use

Related

• Private Federated Learning in Gboard
• Federated Learning of Gboard Language Models with Differential Privacy
• Two Models are Better than One: Federated Learning Is Not Private For Google GBoard Next Word Prediction
• Prompt Public Large Language Models to Synthesize Data for Private On-device Applications
• Privacy-Aware Decoding: Mitigating Privacy Leakage of Large Language Models in Retrieval-Augmented Generation