Beyond Memorization: Violating Privacy Via Inference with Large Language Models
2023/10/11 by Robin Staab, Mark Vero, Staab, Robin +5 · 7 voices · 62 citations
Computer Science · Psychology · Social Sciences · #Artificial intelligence #Cognitive psychology #Computer science #Computer security #Construct (python library) #Data science #Inference #Internet privacy #Memorization #Personally identifiable information #Privacy, Security, and Data Protection #Privacy-Preserving Technologies in Data #Psychology
paper · pdf · doi:10.48550/arxiv.2310.07298
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2023/10/11 · openalex created_date 2023/10/13 · openalex updated_date 2026/07/28
Abstract
Current privacy research on large language models (LLMs) primarily focuses on the issue of extracting memorized training data. At the same time, models' inference capabilities have increased drastically. This raises the key question of whether current LLMs could violate individuals' privacy by inferring personal attributes from text given at inference time. In this work, we present the first comprehensive study on the capabilities of pretrained LLMs to infer personal attributes from text. We construct a dataset consisting of real Reddit profiles, and show that current LLMs can infer a wide range of personal attributes (e.g., location, income, sex), achieving up to 85% top-1 and 95% top-3 accuracy at a fraction of the cost (100×) and time (240×) required by humans. As people increasingly interact with LLM-powered chatbots across all aspects of life, we also explore the emerging threat of privacy-invasive chatbots trying to extract personal information through seemingly benign questions. Finally, we show that common mitigations, i.e., text anonymization and model alignment, are currently ineffective at protecting user privacy against LLM inference. Our findings highlight that current LLMs can infer personal data at a previously unattainable scale. In the absence of working defenses, we advocate for a broader discussion around LLM privacy implications beyond memorization, striving for a wider privacy protection.
Cited by
- Playing Along: Learning a Double-Agent Defender for Belief Steering via Theory of Mind
- StabilityBench: Benchmarking Instability in LLMs
- Large-scale online deanonymization with LLMs
- "These cameras are just like the Eye of Sauron": A Sociotechnical Threat Model for AI-Driven Smart Home Devices as Perceived by UK-Based Domestic Workers
- Are LLMs Smarter Than Chimpanzees? An Evaluation on Perspective Taking and Knowledge State Estimation
- PANOPTICON: A PII-Based Assemblage of Naturalistic Output Tokens for Investigating Privacy Leakage Within LLM Context Window
- Agent Tools Orchestration Leaks More: Dataset, Benchmark, and Mitigation
- ContextLeak: Auditing Leakage in Private In-Context Learning Methods
- SA-ADP: Sensitivity-Aware Adaptive Differential Privacy for Large Language Models
- MultiPriv: Benchmarking Individual-Level Privacy Reasoning in Vision-Language Models
- DRAGON: Guard LLM Unlearning in Context via Negative Detection and Reasoning
- Auditing M-LLMs for Privacy Risks: A Synthetic Benchmark and Evaluation Framework
- Mitigating privacy risks in Retrieval-Augmented Generation via locally private entity perturbation
- Label Smoothing Improves Gradient Ascent in LLM Unlearning
- Energy-Efficient Domain-Specific Artificial Intelligence Models and Agents: Pathways and Paradigms
- Black Box Absorption: LLMs Undermining Innovative Ideas
- CircuitGuard: Mitigating LLM Memorization in RTL Code Generation Against IP Leakage
- From Defender to Devil? Unintended Risk Interactions Induced by LLM Defenses
- Position: Privacy Is Not Just Memorization!
- Operationalizing Data Minimization for Privacy-Preserving LLM Prompting
- Downgrade to Upgrade: Optimizer Simplification Enhances Robustness in LLM Unlearning
- SecInfer: Preventing Prompt Injection via Inference-time Scaling
- OFMU: Optimization-Driven Framework for Machine Unlearning
- Position: Human-Robot Interaction in Embodied Intelligence Demands a Shift From Static Privacy Controls to Dynamic Learning
- RL-Finetuned LLMs for Privacy-Preserving Synthetic Rewriting
- Beyond Data Privacy: New Privacy Risks for Large Language Models
- Beyond PII: How Users Attempt to Estimate and Mitigate Implicit LLM Inference
- LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systems
- Generative Data Refinement: Just Ask for Better Data
- Memorization in Large Language Models in Medicine: Prevalence, Characteristics, and Implications
- A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models
- Standard vs. Modular Sampling: Best Practices for Reliable LLM Unlearning
- Safe-LLaVA: A Privacy-Preserving Vision-Language Dataset and Benchmark for Biometric Safety
- Through Their Eyes: User Perceptions on Sensitive Attribute Inference of Social Media Videos by Visual Language Models
- Protecting Vulnerable Voices: Synthetic Dataset Generation for Self-Disclosure Detection
- Understanding the Supply Chain and Risks of Large Language Model Applications
- Benchmarking LLM Privacy Recognition for Social Robot Decision Making
- What Should LLMs Forget? Quantifying Personal Data in LLMs for Right-to-Be-Forgotten Requests
- The Man Behind the Sound: Demystifying Audio Private Attribute Profiling via Multimodal Large Language Model Agents
- The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigation
- DP-Fusion: Token-Level Differentially Private Inference for Large Language Models
- SoK: The Privacy Paradox of Large Language Models: Advancements, Privacy Risks, and Mitigation
- Beyond Frequency: The Role of Redundancy in Large Language Model Memorization
- Privacy Reasoning in Ambiguous Contexts
- Malicious LLM-Based Conversational AI Makes Users Reveal Personal Information
- Robustly Improving LLM Fairness in Realistic Settings via Interpretability
- Self-Refining Language Model Anonymizers via Adversarial Distillation
- The End Of Universal Lifelong Identifiers: Identity Systems For The AI Era
- Automated Privacy Information Annotation in Large Language Model Interactions
- The Eye of Sherlock Holmes: Uncovering User Private Attribute Profiling via Vision-Language Model Agentic Framework
- Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI's Real World Effects
- Shared Path: Unraveling Memorization in Multilingual LLMs through Language Similarities
- Keep Security! Benchmarking Security Policy Preservation in Large Language Model Contexts Against Indirect Attacks in Question Answering
- Can Large Language Models Really Recognize Your Name?
- GUARD: Generation-time LLM Unlearning via Adaptive Restriction and Detection
- A Comprehensive Analysis of Large Language Model Outputs: Similarity, Diversity, and Bias
- Security of Internet of Agents: Attacks and Countermeasures
- Dependency-Aware Privacy for Multi-turn Agents
- A Survey on Privacy Risks and Protection in Large Language Models
- Can LLMs Infer Conversational Agent Users' Personality Traits from Chat History?
- Beyond Refusal: Probing the Limits of Agentic Self-Correction for Semantic Sensitive Information
- DualOptim: Enhancing Efficacy and Stability in Machine Unlearning with Dual Optimizers
Discussions
- Beyond Memorization: Violating privacy via inference with LLMs [hn, 127 points, 80 comments]
- 👀 KI-Chatbots wissen viel mehr von uns, als wir glauben. Eine neue Studie (Vechev et al. 2023) zeigt, dass Large Language Models eine erschreckende Menge an persönlichen Informationen über Benutzer:i [bsky, 6 points, 0 comments]
- Beyond Memorization: Violating Privacy via Inference with Large Language Models [hn, 3 points, 0 comments]
- Beyond Memorization: Violating Privacy via Inference with Large Language Models [hn, 2 points, 0 comments]
- Beyond Memorization: Violating Privacy via Inference with LLMs [pdf] [hn, 1 points, 0 comments]
- 👀 arxiv.org/abs/2310.07298 [bsky, 0 points, 1 comments]
- Using LLM to identify users by parsing data, nothing new, still scary: arxiv.org/abs/2310.07298 [bsky, 0 points, 0 comments]
Related