Differentially Private Parameter-Efficient Fine-tuning for Large ASR Models

2024/10/02 by Liu, Hongbin, Wang, Lun, Thakkar, Om +2 · 1 citation
#Cryptography and Security (cs.CR) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2410.01948

Abstract

Large ASR models can inadvertently leak sensitive information, which can be mitigated by formal privacy measures like differential privacy (DP). However, traditional DP training is computationally expensive, and can hurt model performance. Our study explores DP parameter-efficient fine-tuning as a way to mitigate privacy risks with smaller computation and performance costs for ASR models. Through extensive experimentation and progressive optimization, we achieve 4.6%/8.1% word error rate on LibriSpeech clean/other test-sets, setting a new performance benchmark while maintaining (10, 3.52e-6)-DP in fine-tuning a large ASR model with over 600M parameters.

Cited by

• The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigation

Related

• Training Large ASR Encoders with Differential Privacy
• Efficient and Private: Memorisation under differentially private parameter-efficient fine-tuning in language models
• Can Differentially Private Fine-tuning LLMs Protect Against Privacy Attacks?
• Revisiting Privacy, Utility, and Efficiency Trade-offs when Fine-Tuning Large Language Models
• Parameter-Efficient Fine-Tuning with Differential Privacy for Robust Instruction Adaptation in Large Language Models