"Power of Words": Stealthy and Adaptive Private Information Elicitation via LLM Communication Strategies

2025/11/15 by Shuning Zhang, Zhang, Shuning, Jiaqi Bai +9
Computer Science · Medicine · Social Sciences · #AI in Service Interactions #Artificial Intelligence in Healthcare and Education #Ethics and Social Impacts of AI #FOS: Computer and information sciences #Human-Computer Interaction (cs.HC)

paper · pdf · doi:10.48550/arxiv.2511.11961

openalex publication_date 2025/11/15 · openalex created_date 2025/11/19 · openalex updated_date 2026/07/28

Abstract

While communication strategies of Large Language Models (LLMs) are crucial for human-LLM interactions, they can also be weaponized to elicit private information, yet such stealthy attacks remain under-explored. This paper introduces the first adaptive attack framework for stealthy and targeted private information elicitation via communication strategies. Our framework operates in a dynamic closed-loop: it first performs real-time psychological profiling of the users' state, then adaptively selects an optimized communication strategy, and finally maintains stealthiness through prompt-based rewriting. We validated this framework through a user study (N=84), demonstrating its generalizability across 3 distinct LLMs and 3 scenarios. The targeted attacks achieved a 205.4% increase in eliciting specific targeted information compared to stealthy interactions without strategies. Even stealthy interactions without specific strategies successfully elicited private information in 54.8% cases. Notably, users not only failed to detect the manipulation but paradoxically rated the attacking chatbot as more empathetic and trustworthy. Finally, we advocate for mitigations, encouraging developers to integrate adaptive, just-in-time alerts, users to build literacy against specific manipulative tactics, and regulators to define clear ethical boundaries distinguishing benign persuasion from coercion.

Citations

• Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Tasks
• Malicious LLM-Based Conversational AI Makes Users Reveal Personal Information
• Adanonymizer: Interactively Navigating and Balancing the Duality of Privacy and Output Performance in Human-LLM Interaction
• AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs
• Trust No Bot: Discovering Personal Disclosures in Human-LLM Conversations in the Wild
• WildChat: 1M ChatGPT Interaction Logs in the Wild
• Reading Users' Minds from What They Say: An Investigation into LLM-based Empathic Mental Inference
• BSPA: Exploring Black-box Stealthy Prompt Attacks against Image Generators
• Large Language Models Meet Open-World Intent Discovery and Recognition: An Evaluation of ChatGPT
• AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
• Knowledge Unlearning for Mitigating Privacy Risks in Language Models
• Do Large Language Models know what humans know?
• Differentially Private Decoding in Large Language Models
• Quantifying Memorization Across Neural Language Models
• Ethical and social risks of harm from Language Models
• An Efficient DP-SGD Mechanism for Large Scale NLP Models
• On the (In)Feasibility of Attribute Inference Attacks on Machine\n Learning Models
• Membership Inference Attacks against Machine Learning Models
• Your Voice Assistant is Mine: How to Abuse Speakers to Steal Information and Control Your Phone

Related

• Attack the Messages, Not the Agents: A Multi-round Adaptive Stealthy Tampering Framework for LLM-MAS
• Defending Against Social Engineering Attacks in the Age of LLMs
• Personalized Attacks of Social Engineering in Multi-turn Conversations: LLM Agents for Simulation and Detection
• Who's the Mole? Modeling and Detecting Intention-Hiding Malicious Agents in LLM-Based Multi-Agent Systems
• Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communities