Federated Learning for Emoji Prediction in a Mobile Keyboard

2019/06/11 by Ramaswamy, Swaroop, Mathews, Rajiv, Rao, Kanishka +1 · 4 citations
#Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.1906.04329

Abstract

We show that a word-level recurrent neural network can predict emoji from text typed on a mobile keyboard. We demonstrate the usefulness of transfer learning for predicting emoji by pretraining the model using a language modeling task. We also propose mechanisms to trigger emoji and tune the diversity of candidates. The model is trained using a distributed on-device learning framework called federated learning. The federated model is shown to achieve better performance than a server-trained model. This work demonstrates the feasibility of using federated learning to train production-quality models for natural language understanding tasks while keeping users' data on their devices.

Cited by

• Minimizing Layerwise Activation Norm Improves Generalization in Federated Learning
• FedSCAl: Leveraging Server and Client Alignment for Unsupervised Federated Source-Free Domain Adaptation
• Prompt Estimation from Prototypes for Federated Prompt Tuning of Vision Transformers
• Understanding Textual Emotion Through Emoji Prediction

Related

• Federated Learning Based Multilingual Emoji Prediction In Clean and Attack Scenarios
• Federated Learning for Mobile Keyboard Prediction
• NTUA-SLP at SemEval-2018 Task 2: Predicting Emojis using RNNs with\n Context-aware Attention
• UMDSub at SemEval-2018 Task 2: Multilingual Emoji Prediction Multi-channel Convolutional Neural Network on Subword Embedding
• Understanding Textual Emotion Through Emoji Prediction