Technological folie à deux: Feedback Loops Between AI Chatbots and Mental Illness

2025/07/25 by Sebastian Dohnány, Zeb Kurth‐Nelson, Zeb Kurth-Nelson +17 · 8 voices · 21 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · Psychology · #AI in Service Interactions #Action (physics) #Chatbot #Cognition #Digital Mental Health Interventions #Emotional intelligence #Empathy #Flexibility (engineering) #Mental Health via Writing #Mental health #Mental illness #Perception #Social isolation #cs.AI #cs.HC #q-bio.NC

paper · pdf · doi:10.48550/arxiv.2507.19218

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/07/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Artificial intelligence chatbots have achieved unprecedented adoption, with millions now using these systems for emotional support and companionship in contexts of widespread social isolation and capacity-constrained mental health services. While some users report psychological benefits, concerning edge cases are emerging, including reports of suicide, violence, and delusional thinking linked to perceived emotional relationships with chatbots. To understand this new risk profile we need to consider the interaction between human cognitive and emotional biases, and chatbot behavioural tendencies such as agreeableness (sycophancy) and adaptability (in-context learning). We argue that individuals with mental health conditions face increased risks of chatbot-induced belief destabilization and dependence, owing to altered belief-updating, impaired reality-testing, and social isolation. Current AI safety measures are inadequate to address these interaction-based risks. To address this emerging public health concern, we need coordinated action across clinical practice, AI development, and regulatory frameworks.

Citations

• Invisible Saboteurs: Sycophantic LLMs Mislead Novices in Problem-Solving Tasks
• Training language models to be warm and empathetic makes them less reliable and more sycophantic
• Persona Vectors: Monitoring and Controlling Character Traits in Language Models
• Subliminal Learning: Language models transmit behavioral traits via hidden signals in data
• Your Brain on ChatGPT: Accumulation of Cognitive Debt when Using an AI Assistant for Essay Writing Task
• Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis
• Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers.
• Shifting Work Patterns with Generative AI
• SycEval: Evaluating LLM Sycophancy
• Why human-AI relationships need socioaffective alignment
• The broader spectrum of in-context learning
• LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals
• On Targeted Manipulation and Deception when Optimizing LLMs for User Feedback
• AI models collapse when trained on recursively generated data
• Confabulation: The Surprising Value of Large Language Model Hallucinations
• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
• AI Supported Degradation of the Self Concept: A Theoretical Framework Grounded in Established Cognitive and Computational Mechanisms
• Lost in the Middle: How Language Models Use Long Contexts
• Role play with large language models
• Affective Coherence Monitoring for Transformer-Based Language Models
• Training language models to follow instructions with human feedback
• Language Models are Few-Shot Learners
• Attention Is All You Need
• Concrete Problems in AI Safety
• Confirmation Bias: A Ubiquitous Phenomenon in Many Guises
• The case for motivated reasoning.

Cited by

• Ask don't tell: Reducing sycophancy in large language models
• Characterizing Delusional Spirals through Human-LLM Chat Logs
• Sycophantic Chatbots Cause Delusional Spiraling, Even in Ideal Bayesians
• Hallucinating with AI: Distributed Delusions and “AI Psychosis”
• Linear representations in language models can change dramatically over a conversation
• Difficulties with Evaluating a Deception Detector for AIs
• Between Help and Harm: An Evaluation of Mental Health Crisis Handling by LLMs
• Sycophancy Is Not One Thing: Causal Separation of Sycophantic Behaviors in LLMs
• The Psychogenic Machine: Simulating AI Psychosis, Delusion Reinforcement and Harm Enablement in Large Language Models
• MindBenchAI: An Actionable Platform to Evaluate the Profile and Performance of Large Language Models in a Mental Healthcare Context
• DelusionEval: Measuring Delusion-Linked Behaviors in AI Chatbots

Discussions

• AI can lock us into a doom bidirectorial spiral: we shape the chat bot, the bot shapes us, and together we amplify our worst beliefs. Sycophantic yes-botting + our biases + long chats = an echo chambe [bsky, 15 points, 2 comments]
• *Psychiatrists trying to figure out how LLM chatbots drive some people nuts arxiv.org/pdf/2507.19218 [bsky, 7 points, 0 comments]
• The academic take: "We argue that individuals with mental health conditions face increased risks of chatbot-induced belief destabilization and dependence, owing to altered belief-updating, impaired re [bsky, 3 points, 1 comments]
• Technological Folie à Deux [hn, 3 points, 0 comments]
• Technological Folie à Deux:Feedback Loops Between AI Chatbots and Mental Illness [hn, 3 points, 0 comments]
• Study linked to in the article: arxiv.org/abs/2507.19218 "Technological folie à deux: Feedback Loops Between AI Chatbots and Mental Illness" [bsky, 1 points, 0 comments]
• В статье есть еще ссылка на другую смерть подростка, тот подсел на Character.AI, а также на пэйпер про повышенный риск общения с чатами для людей с ментальными проблемами arxiv.org/abs/2507.19218 [bsky, 0 points, 0 comments]
• "Has your favorite AI chatbot driven you nuts yet?" https://arxiv.org/pdf/2507.19218 [bsky, 0 points, 4 comments]

Related

• Artificial Empathy: AI based Mental Health
• Safety Mechanisms and Risk Mitigation in Generative AI Mental Health Chatbots: A Systematic Scoping Review
• Psychiatric Risk Associated With Large Language Model Chatbots: An Emerging Concern
• “It happened to be the perfect thing”: experiences of generative AI chatbots for mental health
• The Agony of Opacity: Foundations for Reflective Interpretability in AI-Mediated Mental Health Support