Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs

2024/03/08 by Xuhui Zhou, Zhe Su, Zhou, Xuhui +7 · 1 voice · 9 citations
Computer Science · Social Sciences · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Law #Computation and Language (cs.CL) #FOS: Computer and information sciences #cs.AI #cs.CL

paper · pdf · doi:10.48550/arxiv.2403.05020

openalex publication_date 2024/03/08 · arxiv published 2024/03/08 · openalex created_date 2024/03/13 · arxiv updated 2024/10/04 · openalex updated_date 2026/07/28

Abstract

Recent advances in large language models (LLM) have enabled richer social simulations, allowing for the study of various social phenomena. However, most recent work has used a more omniscient perspective on these simulations (e.g., single LLM to generate all interlocutors), which is fundamentally at odds with the non-omniscient, information asymmetric interactions that involve humans and AI agents in the real world. To examine these differences, we develop an evaluation framework to simulate social interactions with LLMs in various settings (omniscient, non-omniscient). Our experiments show that LLMs perform better in unrealistic, omniscient simulation settings but struggle in ones that more accurately reflect real-world conditions with information asymmetry. Our findings indicate that addressing information asymmetry remains a fundamental challenge for LLM-based agents.

Cited by

• MASim: Multilingual Agent-Based Simulation for Social Science
• The Imperfect Learner: Incorporating Developmental Trajectories in Memory-based Student Simulation
• Communication and Verification in LLM Agents towards Collaboration under Information Asymmetry
• Learning Time-Varying Turn-Taking Behavior in Group Conversations
• Generative Value Conflicts Reveal LLM Priorities
• Virus Infection Attack on LLMs: Your Poisoning Can Spread "VIA" Synthetic Data
• What Makes LLM Agent Simulations Useful for Policy? Insights From an Iterative Design Engagement in Emergency Preparedness
• The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societies
• Social World Models

Discussions

• My PhD student Xuhui Zhou will present "Is This the Real Life? Is This Just Fantasy? The Misleading Success of Simulating Social Interactions With LLMs" arxiv.org/abs/2403.05020 [bsky, 4 points, 1 comments]

Related

• LLM Social Simulations Are a Promising Research Method
• Social Simulations with Large Language Model Risk Utopian Illusion
• LLM-Based Social Simulations Require a Boundary
• LLM-Augmented Agent-Based Modelling for Social Simulations: Challenges and Opportunities
• Integrating LLM in Agent-Based Social Simulation: Opportunities and Challenges