Public Data Assisted Differentially Private In-Context Learning

2025/09/13 by Joo, Seongho, Hyukhun Koh, Koh, Hyukhun +2
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning and Data Classification #Privacy-Preserving Technologies in Data

paper · pdf · doi:10.48550/arxiv.2509.10932

openalex publication_date 2025/09/13 · openalex created_date 2025/10/12 · openalex updated_date 2026/07/28

Abstract

In-context learning (ICL) in Large Language Models (LLMs) has shown remarkable performance across various tasks without requiring fine-tuning. However, recent studies have highlighted the risk of private data leakage through the prompt in ICL, especially when LLMs are exposed to malicious attacks. While differential privacy (DP) provides strong privacy guarantees, it often significantly reduces the utility of in-context learning (ICL). To address this challenge, we incorporate task-related public data into the ICL framework while maintaining the DP guarantee. Based on this approach, we propose a private in-context learning algorithm that effectively balances privacy protection and model utility. Through experiments, we demonstrate that our approach significantly improves the utility of private ICL with the assistance of public data. Additionally, we show that our method is robust against membership inference attacks, demonstrating empirical privacy protection.

Citations

• Membership Inference Attacks Against In-Context Learning
• Text Sanitization Beyond Specific Domains: Zero-Shot Redaction & Substitution with Large Language Models
• Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generation
• Backdoor Attacks for In-Context Learning with Language Models
• DPM: Clustering Sensitive Data through Separation
• Selective Pre-training for Private Fine-tuning
• ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge
• A Survey on In-context Learning
• Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
• Submix: Practical Private Prediction for Large-Scale Language Models
• Differentially Private Fine-tuning of Language Models
• Large Language Models Can Be Strong Differentially Private Learners
• Language Models are Few-Shot Learners
• Privacy Amplification by Subsampling: Tight Analyses via Couplings and Divergences
• Scalable Private Learning with PATE
• Membership Inference Attacks against Machine Learning Models
• Concentrated Differential Privacy: Simplifications, Extensions, and Lower Bounds
• DialogSum: A Real-Life Scenario Dialogue Summarization Dataset

Related

• Privacy-Preserving In-Context Learning for Large Language Models
• Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generation
• Locally Differentially Private In-Context Learning
• Privacy Preserving In-Context-Learning Framework for Large Language Models
• SnapAudit: Active Auditing of Differentially Private In-Context Learning via Snapshot-Based Simulation