User Privacy and Large Language Models: An Analysis of Frontier Developers' Privacy Policies

2025/09/05 by Jennifer King, Kevin Klyman, King, Jennifer +7 · 14 voices · 7 citations
Computer Science · Social Sciences · #Privacy, Security, and Data Protection #Privacy-Preserving Technologies in Data #cs.AI #cs.CR #cs.CY

paper · pdf · doi:10.48550/arxiv.2509.05382

openalex publication_date 2025/09/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Hundreds of millions of people now regularly interact with large language models via chatbots. Model developers are eager to acquire new sources of high-quality training data as they race to improve model capabilities and win market share. This paper analyzes the privacy policies of six U.S. frontier AI developers to understand how they use their users' chats to train models. Drawing primarily on the California Consumer Privacy Act, we develop a novel qualitative coding schema that we apply to each developer's relevant privacy policies to compare data collection and use practices across the six companies. We find that all six developers appear to employ their users' chat data to train and improve their models by default, and that some retain this data indefinitely. Developers may collect and train on personal information disclosed in chats, including sensitive information such as biometric and health data, as well as files uploaded by users. Four of the six companies we examined appear to include children's chat data for model training, as well as customer data from other products. On the whole, developers' privacy policies often lack essential information about their practices, highlighting the need for greater transparency and accountability. We address the implications of users' lack of consent for the use of their chat data for model training, data security issues arising from indefinite chat data retention, and training on children's chat data. We conclude by providing recommendations to policymakers and developers to address the data privacy challenges posed by LLM-powered chatbots.

Citations

• Consent in Crisis: The Rapid Decline of the AI Data Commons
• Trust No Bot: Discovering Personal Disclosures in Human-LLM Conversations in the Wild
• Scalable Extraction of Training Data from (Production) Language Models
• Model Cards for Model Reporting

Cited by

• An approach to systemic risks of AI through the lens of emergence, collective action problems, and externalities
• Engagement-Optimized Care: When LLMs become Mental Health Infrastructure
• Terms of (Ab)Use: An Analysis of GenAI Services
• Taxing Artificial Intelligence
• The 2025 Foundation Model Transparency Index
• A Longitudinal Measurement of Privacy Policy Evolution for Large Language Models
• Black Box Absorption: LLMs Undermining Innovative Ideas
• Ask ChatGPT: Caveats and Mitigations for Individual Users of AI Chatbots

Discussions

• All the major AI companies are training on your chats with them, Stanford's @kingjen.bsky.social finds arxiv.org/abs/2509.05382 [bsky, 40 points, 2 comments]
• As of 2025, this analysis of privacy policies indicates that every major AI company uses your private conversations to train their models by default. Every prompt, file, photo, personal detail: all of [bsky, 15 points, 3 comments]
• No one should be surprised that AI tech companies are using your chat history on their platform to train new models arxiv.org/abs/2509.05382 #privacy #security #ethics #ai #llms [bsky, 8 points, 1 comments]
• Paper: User Privacy and Large Language Models: An Analysis of Frontier Developers' Privacy Policies arxiv.org/abs/2509.05382 [bsky, 5 points, 0 comments]
• Im just putting this link here so that I can remember to read it later arxiv.org/abs/2509.05382 [bsky, 4 points, 0 comments]
• User Privacy and LLMs: An Analysis of Frontier Developers' Privacy Policies [hn, 3 points, 0 comments]
• "We find that all six developers appear to employ their users' chat data to train and improve their models by default, and that some retain this data indefinitely" oh well arxiv.org/abs/2509.05382 [bsky, 3 points, 0 comments]
• I absolutely hear your frustration with this. Peer review is hard enough as it is. Except loading an abstract into a Gen AI app is highly problematic: What might be original work just became part of a [bsky, 3 points, 2 comments]
• User Privacy and Large Language Models: An Analysis of Frontier Developers' Privacy Policies arxiv.org/abs/2509.05382 [bsky, 1 points, 0 comments]
• User Privacy: An Analysis of Frontier LLM Privacy Policies (2025) [hn, 1 points, 0 comments]
• @timnitgebru.bsky.social @mmitchell.bsky.social very lost irony by poster's succinct précis you should read based on 1 1/2 yo AI paper below: Bad place post x.com/heygurisingh... Paper it's based upon [bsky, 0 points, 0 comments]
• seastersjones Stanford analysis: arxiv.org/abs/2509.05382 [bsky, 0 points, 1 comments]
• arxiv.org/abs/2509.05382 [bsky, 0 points, 0 comments]
• Study: Chatbots/LLMs pose serious new privacy risks, deepen consumer exploitation. Amazon, Meta & OpenAI leave children especially exposed. AI hype is not worth “the loss of privacy and increased risk [bsky, 0 points, 0 comments]

Related

• SoK: The Privacy Paradox of Large Language Models: Advancements, Privacy Risks, and Mitigation
• Human-Centered Privacy Research in the Age of Large Language Models
• Privacy Issues in Large Language Models: A Survey
• User Privacy Harms and Risks in Conversational AI: A Proposed Framework
• Guarding Your Conversations: Privacy Gatekeepers for Secure Interactions with Cloud-Based AI Models