Generative Knowledge Transfer for Neural Language Models

2016/08/14 by Sungho Shin, Shin, Sungho, Kyuyeon Hwang +3 · 1 citation
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Privacy-Preserving Technologies in Data #Speech Recognition and Synthesis #Topic Modeling #cs.LG

paper · pdf · doi:10.48550/arxiv.1608.04077

openalex publication_date 2016/08/14 · arxiv created 2017/02/28 · arxiv updated 2017/03/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we propose a generative knowledge transfer technique that trains an RNN based language model (student network) using text and output probabilities generated from a previously trained RNN (teacher network). The text generation can be conducted by either the teacher or the student network. We can also improve the performance by taking the ensemble of soft labels obtained from multiple teacher networks. This method can be used for privacy conscious language model adaptation because no user data is directly used for training. Especially, when the soft labels of multiple devices are aggregated via a trusted third party, we can expect very strong privacy protection.

Citations

• ADADELTA: An Adaptive Learning Rate Method
• FitNets: Hints for Thin Deep Nets
• Distilling the Knowledge in a Neural Network
• DRAW: A Recurrent Neural Network For Image Generation
• Long Short-Term Memory
• Do Deep Nets Really Need to be Deep?
• A Survey on Transfer Learning
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

Cited by

• Image Generation From Small Datasets via Batch Statistics Adaptation

Related

• Efficient Transfer Learning Schemes for Personalized Language Modeling using Recurrent Neural Network
• Social Learning: Towards Collaborative Learning with Large Language Models
• Pretraining with Artificial Language: Studying Transferable Knowledge in Language Models
• GenDistiller: Distilling Pre-trained Language Models based on Generative Models
• Variational Information Distillation for Knowledge Transfer