Evaluating large language models in theory of mind tasks
2023/02/04 by Michal Kosinski, Michał Kosiński · 6 voices · 64 citations
Psychology · Computer Science · Social Sciences · #Child and Animal Learning Development #Topic Modeling #Language and cultural evolution
paper · pdf · doi:10.1073/pnas.2405460121
Abstract
Eleven large language models (LLMs) were assessed using 40 bespoke false-belief tasks, considered a gold standard in testing theory of mind (ToM) in humans. Each task included a false-belief scenario, three closely matched true-belief control scenarios, and the reversed versions of all four. An LLM had to solve all eight scenarios to solve a single task. Older models solved no tasks; Generative Pre-trained Transformer (GPT)-3-davinci-003 (from November 2022) and ChatGPT-3.5-turbo (from March 2023) solved 20% of the tasks; ChatGPT-4 (from June 2023) solved 75% of the tasks, matching the performance of 6-y-old children observed in past studies. We explore the potential interpretation of these results, including the intriguing possibility that ToM-like ability, previously considered unique to humans, may have emerged as an unintended by-product of LLMs' improving language skills. Regardless of how we interpret these outcomes, they signify the advent of more powerful and socially skilled AI-with profound positive and negative implications.
Cited by
- Playing Along: Learning a Double-Agent Defender for Belief Steering via Theory of Mind
- Analyzing the Ethical Logic of Eight Large Language Models
- The Severance Problem: LLMs are Unaware of the Person Beyond the Prompt
- The Theory of Mind Utility: Formal Specification of a Mentalizing Mechanism
- MeetingToM: Evaluating Multimodal LLMs on Theory-of-Mind Reasoning in Multi-Party Meetings
- Belief-reality separation lives in routing over a shared value slot in language models
- MafiaScope: Non-Invasive, Time-Resolved Belief Probing for LLM Agents in Social Deduction Games
- Collaborative Spatial Learning with Multi-LLM Agents in Networked Social Experiments
- Robust Critics: Defending LLMs Against Multi-Turn Attacks
- Are LLMs Smarter Than Chimpanzees? An Evaluation on Perspective Taking and Knowledge State Estimation
- How Human is AI? Examining the Impact of Emotional Prompts on Artificial and Human and Responsiveness
- Are Large Language Models Sensitive to the Motives Behind Communication?
- Artificial Phantasia: Emergent Mental Imagery in Large Language Models
- Large Language Models Do Not Simulate Human Psychology
- The Homogenizing Effect of Large Language Models on Human Expression and Thought
- Large Language Models are Near-Optimal Decision-Makers with a Non-Human Learning Behavior
- LLM Social Simulations Are a Promising Research Method
- Re-evaluating Theory of Mind evaluation in large language models
- Observer, Not Player: Simulating Theory of Mind in LLMs through Game Observation
- Are Vision Language Models Cross-Cultural Theory of Mind Reasoners?
- Few-Shot Inference of Human Perceptions of Robot Performance in Social Navigation Scenarios
- Can GPT replace human raters? Validity and reliability of machine-generated norms for metaphors
- Large Language Models have Chain-of-Affect
- Single-Agent Scaling Fails Multi-Agent Intelligence: Towards Foundation Models with Native Multi-Agent Intelligence
- Love First, Know Later: Persona-Based Romantic Compatibility Through LLM Text World Engines
- Us-vs-Them bias in Large Language Models
- Tacit Bidder-Side Collusion: Artificial Intelligence in Dynamic Auctions
- Mind the Motions: Benchmarking Theory-of-Mind in Everyday Body Language
- From Passive to Persuasive: Localized Activation Injection for Empathy and Negotiation
- PerspAct: Enhancing LLM Situated Collaboration Skills through Perspective Taking and Active Vision
- Characterizing AI Manipulation Risks in Brazilian YouTube Climate Discourse
- Cognitive Convergence: Deep Similarities Between Large Language Models and Human Cognition
- Large Language Models in Human Subject Research, and the Presence of Idiosyncratic Human Behaviors
- Self-Reflection Protects Behavior from Volatile Beliefs Linked to Paranoia
- The Rise of AI Agent Communities: Large-Scale Analysis of Discourse and Interaction on Moltbook
- Social Simulations with Large Language Model Risk Utopian Illusion
- A Design Science Blueprint for an Orchestrated AI Assistant in Doctoral Supervision
- DPRF: A Generalizable Dynamic Persona Refinement Framework for Optimizing Behavior Alignment Between Personalized LLM Role-Playing Agents and Humans
- Doing Things with Words: Rethinking Theory of Mind Simulation in Large Language Models
- Do You Get the Hint? Benchmarking LLMs on the Board Game Concept
- Circuit Distillation
- The evolving field of digital mental health: current evidence and implementation issues for smartphone apps, generative artificial intelligence, and virtual reality
- Infusing Theory of Mind into Socially Intelligent LLM Agents
- GPT (Generative Pre-Trained Transformer)— A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions
- The Philosophy of Language Models
- Not Everyone Wins with LLMs: Behavioral Patterns and Pedagogical Implications in AI-assisted Data Analysis
- Persuading large language models to comply with objectionable requests
- A large-scale evaluation of commonsense knowledge in humans and large language models
- The homogenizing effect of large language models on human expression and thought
- On the creativity of large language models
- LVLMs are Bad at Overhearing Human Referential Communication
- Preservation of Language Understanding Capabilities in Speech-aware Large Language Models
- One Model, Two Minds: A Context-Gated Graph Learner that Recreates Human Biases
- Drivel-ology: Challenging LLMs with Interpreting Nonsense with Depth
- The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMs
- Do LLMs Adhere to Label Definitions? Examining Their Receptivity to External Label Definitions
- LLMs and their Limited Theory of Mind: Evaluating Mental State Annotations in Situated Dialogue
- Bridging Minds and Machines: Toward an Integration of AI and Cognitive Science
- Who Sees What? Structured Thought-Action Sequences for Epistemic Reasoning in LLMs
- Exploring Large Language Model Agents for Piloting Social Experiments
- LLMs for Resource Allocation: A Participatory Budgeting Approach to Inferring Preferences
- VirtLab: An AI-Powered System for Flexible, Customizable, and Large-scale Team Simulations
- Knowing you know nothing in the age of generative AI
- Evaluating Theory of Mind and Internal Beliefs in LLM-Based Multi-Agent Systems
Discussions
- Theory of Mind May Have Spontaneously Emerged in Large Language Models [hn, 170 points, 309 comments]
- Theory of Mind in LLMs https://arxiv.org/abs/2302.02083 [bsky, 5 points, 0 comments]
- AGI輪読会でこちら。心の理論(ToM)を人間と同じように質問形式で評価するのは無理があって、やっぱり叙述トリックとかそっちの方に振ったほうがいいのでは、と思いました。人間と違うやり方で解いてると主張するなら、評価も変えるべきでしょう。具体的方法はないですが…… arxiv.org/abs/2302.02083 [bsky, 2 points, 0 comments]
- Theory of Mind May Have Spontaneously Emerged in Large Language Models [bsky, 0 points, 0 comments]
- Which preprint? Do you mean this one by Kosinski, where it already did pretty well? This new paper references it and shows how some changes in tests that fool LLMs do so with humans too, contrary to [bsky, 0 points, 1 comments]
- これまでヒト特有の能力と考えられてきた心の理論が大規模言語モデル (LLM) において自然発生的に出現したのではないか、という仮説を検証した研究。40 個の誤信念課題を複数の LLM (GPT-3-davinci-001 から ChatGPT-4) に実施した結果、ChatGPT-4 では 90% クリアして、ヒトの 7 歳と同程度だった。LLM の言語能力向上は心の理論を自然発生させた可能性を示 [bsky, 0 points, 1 comments]
Related