Hush! Protecting Secrets During Model Training: An Indistinguishability Approach

2025/05/30 by Arun Ganesh, Ganesh, Arun, Brendan McMahan +7 · 1 citation
Computer Science · #Cloud Data Security Solutions #Cryptography and Security (cs.CR) #Digital and Cyber Forensics #FOS: Computer and information sciences #Security and Verification in Computing

paper · pdf · doi:10.48550/arxiv.2506.00201

openalex publication_date 2025/05/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We consider the problem of secret protection, in which a business or organization wishes to train a model on their own data, while attempting to not leak secrets potentially contained in that data via the model. The standard method for training models to avoid memorization of secret information is via differential privacy (DP). However, DP requires a large loss in utility or a large dataset to achieve its strict privacy definition, which may be unnecessary in our setting where the data curator and data owner are the same entity. We propose an alternate definition of secret protection that instead of targeting DP, instead targets a bound on the posterior probability of secret reconstruction. We then propose and empirically evaluate an algorithm for model training with this secret protection definition. Our algorithm solves a linear program to assign weights to examples based on the desired per-secret protections, and then performs Poisson sampling using these weights. We show our algorithm significantly outperforms the baseline of running DP-SGD on the whole dataset.

Citations

• It's My Data Too: Private ML for Datasets with Multi-User Training Examples
• Fine-Tuning Large Language Models with User-Level Differential Privacy
• Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuning
• Scalable Extraction of Training Data from (Production) Language Models
• Bounding Training Data Reconstruction in DP-SGD
• Individual Privacy Accounting for Differentially Private Stochastic Gradient Descent
• Quantifying Memorization Across Neural Language Models
• Reconstructing Training Data with Informed Adversaries
• Optimal Accounting of Differential Privacy via Characteristic Function
• Extracting Training Data from Large Language Models
• On the Use of ArXiv as a Dataset
• Protection Against Reconstruction and Its Applications in Private Federated Learning
• Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• Secret-Protected Evolution for Differentially Private Synthetic Text Generation

Related

• Provably Confidential Language Modelling
• Too Good to be True? Turn Any Model Differentially Private With DP-Weights
• SecretGen: Privacy Recovery on Pre-Trained Models via Distribution Discrimination
• Three Tools for Practical Differential Privacy
• Accurate, Explainable, and Private Models: Providing Recourse While Minimizing Training Data Leakage