Natural Language Understanding with Privacy-Preserving BERT

2021/04/30 by Chen Qu, Weize Kong, Liu Yang +3 · 3 citations
Computer Science · #cs.CL

paper · pdf · doi:10.1145/3459637.3482281

Accepted to CIKM 2021

arxiv created 2021/08/19 · arxiv updated 2021/08/23

Abstract

Privacy preservation remains a key challenge in data mining and Natural Language Understanding (NLU). Previous research shows that the input text or even text embeddings can leak private information. This concern motivates our research on effective privacy preservation approaches for pretrained Language Models (LMs). We investigate the privacy and utility implications of applying dx-privacy, a variant of Local Differential Privacy, to BERT fine-tuning in NLU applications. More importantly, we further propose privacy-adaptive LM pretraining methods and show that our approach can boost the utility of BERT dramatically while retaining the same level of privacy protection. We also quantify the level of privacy preservation and provide guidance on privacy configuration. Our experiments and findings lay the groundwork for future explorations of privacy-preserving NLU with pretrained LMs.

Cited by

• Towards Harnessing the Collaborative Power of Large and Small Models for Domain Tasks
• Anti-adversarial Learning: Desensitizing Prompts for Large Language Models
• DeepInvert: Semi-Supervised Embedding Inversion Against Obfuscated Language Models