VaultGemma: A Differentially Private Gemma Model

2025/10/15 by Sinha, Amer, Mesnard, Thomas, McKenna, Ryan +18 · 4 voices · 2 citations
#Artificial Intelligence (cs.AI) #Cryptography and Security (cs.CR) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2510.15001

Abstract

We introduce VaultGemma 1B, a 1 billion parameter model within the Gemma family, fully trained with differential privacy. Pretrained on the identical data mixture used for the Gemma 2 series, VaultGemma 1B represents a significant step forward in privacy-preserving large language models. We openly release this model to the community

Citations

• Scaling Laws for Differentially Private Language Models
• Scalable DP-SGD: Shuffling vs. Poisson Subsampling
• Gemma 2: Improving Open Language Models at a Practical Size
• Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenon
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
• Emergent and Predictable Memorization in Large Language Models
• Analyzing Leakage of Personally Identifiable Information in Language Models
• Position: Considerations for Differentially Private Learning with Large-Scale Public Pretraining
• Connect the Dots: Tighter Discrete Approximations of Privacy Loss Distributions
• Training Compute-Optimal Large Language Models
• Practical and Private (Deep) Learning without Sampling or Shuffling
• Extracting Training Data from Large Language Models
• Root Mean Square Layer Normalization
• SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing
• Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy

Cited by

• ContextLeak: Auditing Leakage in Private In-Context Learning Methods
• PrivCode: When Code Generation Meets Differential Privacy

Discussions

• VaultGemma: A differentially private LLM (2025) [lobsters, 7 points, 3 comments]
• VaultGemma: A Differentially Private LLM [hn, 3 points, 0 comments]
• VaultGemma: A differentially private LLM https://lobste.rs/s/mihtmk #privacy #ai [bsky, 0 points, 0 comments]
• VaultGemma - A Differentially Private Gemma Model arxiv.org/abs/2510.15001 [bsky, 0 points, 0 comments]

Related

• Learning Differentially Private Recurrent Language Models
• You Are What You Write: Preserving Privacy in the Era of Large Language Models
• Privacy-Preserving Large Language Models: Mechanisms, Applications, and Future Directions
• Fine-Tuning Large Language Models with User-Level Differential Privacy
• Scaling Laws for Differentially Private Language Models