LLM Social Simulations Are a Promising Research Method

2025/04/03 by Jacy Reese Anthis, Ryan Liu, Anthis, Jacy Reese +16 · 2 voices · 74 citations
Social Sciences · #Artificial Intelligence in Law #Computer science #Environmental science #Political science

paper · pdf · doi:10.48550/arxiv.2504.02234

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/04/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Accurate and verifiable large language model (LLM) simulations of human research subjects promise an accessible data source for understanding human behavior and training new AI systems. However, results to date have been limited, and few social scientists have adopted this method. In this position paper, we argue that the promise of LLM social simulations can be achieved by addressing five tractable challenges. We ground our argument in a review of empirical comparisons between LLMs and human research subjects, commentaries on the topic, and related work. We identify promising directions, including context-rich prompting and fine-tuning with social science datasets. We believe that LLM social simulations can already be used for pilot and exploratory studies, and more widespread use may soon be possible with rapidly advancing LLM capabilities. Researchers should prioritize developing conceptual models and iterative evaluations to make the best use of new AI systems.

Citations

• Machine Bias. How Do Generative Language Models Answer Opinion Polls? <sup/>
• Language Models Use Trigonometry to Do Addition
• LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals
• One fish, two fish, but not the whole sea: Alignment reduces language models’ conceptual diversity
• Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse
• Centaur: a foundation model of human cognition
• LLMs achieve adult human performance on higher-order theory of mind tasks
• Automated Social Science: Language Models as Scientist and Subjects
• Thousands of AI Authors on the Future of AI
• Levels of AGI for Operationalizing Progress on the Path to AGI
• AI Supported Degradation of the Self Concept: A Theoretical Framework Grounded in Established Cognitive and Computational Mechanisms
• Consciousness in Artificial Intelligence: Insights from the Science of Consciousness
• Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks
• Role play with large language models
• QLoRA: Efficient Finetuning of Quantized LLMs
• Generative Agents: Interactive Simulacra of Human Behavior
• Sparks of Artificial General Intelligence: Early experiments with GPT-4
• Evaluating large language models in theory of mind tasks
• Affective Coherence Monitoring for Transformer-Based Language Models
• On the Dangers of Stochastic Parrots
• Attention Is All You Need
• Methods of coping with social desirability bias: A review
• I.—COMPUTING MACHINERY AND INTELLIGENCE

Cited by

• Computational Turing Test Reveals Systematic Differences Between Human and AI Language
• Moloch's Bargain: Emergent Misalignment When LLMs Compete for Audiences
• The threat of analytic flexibility in using large language models to simulate human data
• Challenges in Statistics: A Dozen Challenges in Causality and Causal Inference
• Informing AI Policy Assessment using Large-Scale Simulation of Interventions
• Agent-based simulation of online social networks and disinformation
• Statistical realism is not evidence that LLMs can estimate treatment effects in social science experiments
• Generative AI as Digital Representatives in Collective Decision-Making: A Game-Theoretical Approach
• MASim: Multilingual Agent-Based Simulation for Social Science
• Knowing Your Uncertainty -- On the application of LLM in social sciences
• CrowdLLM: Building LLM-Based Digital Populations Augmented with Generative Models
• Agent-Kernel: A MicroKernel Multi-Agent System Framework for Adaptive Social Simulation Powered by LLMs
• Can Intelligent User Interfaces Engage in Philosophical Discussions? A Longitudinal Study of Philosophers' Evolving Perceptions
• Social Perceptions of English Spelling Variation on Twitter: A Comparative Analysis of Human and LLM Responses
• Can Finetuing LLMs on Small Human Samples Increase Heterogeneity, Alignment, and Belief-Action Coherence?
• Generative AI in Sociological Research: State of the Discipline
• A Criminology of Machines
• Graph-Based Alternatives to LLMs for Human Simulation
• Simulating and Experimenting with Social Media Mobilization Using LLM Agents
• Will Scaling Improve Social Simulation with LLMs?
• Persona Generators: Generating Diverse Synthetic Personas for Arbitrary Contexts
• LLM-augmented empirical game theoretic simulation for social-ecological systems
• World Models Should Prioritize the Unification of Physical and Social Dynamics
• Black Box Absorption: LLMs Undermining Innovative Ideas
• See, Think, Act: Online Shopper Behavior Simulation with VLM Agents
• SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors
• Tailored untruths: How personalisation challenges LLM safeguards
• Valid Survey Simulations with Limited Human Data: The Roles of Prompting, Fine-Tuning, and Rectification
• SocioBench: Modeling Human Behavior in Sociological Surveys with Large Language Models
• Scaling Law in LLM Simulated Personality: More Detailed and Realistic Persona Profile Is All You Need
• Simulating Teams with LLM Agents: Interactive 2D Environments for Studying Human-AI Dynamics
• ReviewerToo: Should AI Join The Program Committee? A Look At The Future of Peer Review
• When Machines Meet Each Other: Network Effects and the Strategic Role of History in Multi-Agent AI
• Prototyping Digital Social Spaces through Metaphor-Driven Design: Translating Spatial Concepts into an Interactive Social Simulation
• PolicyPad: Collaborative Prototyping of LLM Policies
• This human study did not involve human subjects: Validating LLM simulations as behavioral evidence
• A large-scale evaluation of commonsense knowledge in humans and large language models
• The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societies
• OnlineMate: An LLM-Based Multi-Agent Companion System for Cognitive Support in Online Learning
• Synthetic Data Generation for Screen Time and App Usage
• Inject, Fork, Compare: Defining an Interaction Vocabulary for Multi-Agent Simulation Platforms
• Programmable Cognitive Bias in Social Agents
• RecoWorld: Building Simulated Environments for Agentic Recommender Systems
• The Language of Approval: Identifying the Drivers of Positive Feedback Online
• HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistants
• Large Language Models as Virtual Survey Respondents: Evaluating Sociodemographic Response Generation
• Are LLM Agents Behaviorally Coherent? Latent Profiles for Social Simulation
• Noise, Adaptation, and Strategy: Assessing LLM Fidelity in Decision-Making
• Valid Inference with Imperfect Synthetic Data
• Do Machines Think Emotionally? Cognitive Appraisal Analysis of Large Language Models
• Beyond Brainstorming: What Drives High-Quality Scientific Ideas? Lessons from Multi-Agent Collaboration
• Nine Raters, One Index: Carrying LLM Disagreement into Labour-Market Estimates
• Validating Generative Agent-Based Models of Social Norm Enforcement: From Replication to Novel Predictions
• Integrating LLM in Agent-Based Social Simulation: Opportunities and Challenges
• SYNTHIA: Synthetic Yet Naturally Tailored Human-Inspired PersonAs
• Infected Smallville: How Disease Threat Shapes Sociality in LLM Agents
• Multi-Actor Generative Artificial Intelligence as a Game Engine
• Many LLMs Are More Utilitarian Than One
• Epitome: Pioneering an Experimental Platform for AI-Social Science Integration
• LLM-Based Social Simulations Require a Boundary
• Social Scientists on the Role of AI in Research
• Aligning LLM with human travel choices: a persona-based embedding learning approach
• Integrating Counterfactual Simulations with Language Models for Explaining Multi-Agent Behaviour
• Can AI Agents Simulate A/B Test Outcomes? A Validation Framework for Agentic Experimentation
• Seeing Through Deception: Uncovering Misleading Creator Intent in Multimodal News with Vision-Language Models
• MindVote: When AI Meets the Wild West of Social Media Opinion
• Breaking Bad Tokens: Detoxification of LLMs Using Sparse Autoencoders
• Exploring Silicon-Based Societies: An Early Study of the Moltbook Agent Community
• RCTs for Frontier AI Governance: Methodological Challenges and Solutions for Human Uplift Studies
• LLM-Based Educational Simulation: Evaluating Temporal Student Persona Stability Across ADHD Profiles
• A closer look at how large language models trust humans: patterns and biases
• Computational Basis of LLM's Decision Making in Social Simulation
• The Judgment-Consequence Gap: LLM Moral Reasoning in Healthcare Decisions
• SocioVerse: A World Model for Social Simulation Powered by LLM Agents and A Pool of 10 Million Real-World Users

Discussions

• Should we use LLMs 🤖 to simulate human research subjects 🧑? In our new preprint, we argue sims can augment human studies to scale up social science as AI technology accelerates. We identify 5 tracta [bsky, 20 points, 3 comments]
• https://bsky.app/profile/matthewdeverna.com/post/3lm3rtd43gk2h [bsky, 1 points, 0 comments]

Related

• LLM-based Human Simulations Have Not Yet Been Reliable
• LLM-Based Social Simulations Require a Boundary
• LLM-Augmented Agent-Based Modelling for Social Simulations: Challenges and Opportunities
• Integrating LLM in Agent-Based Social Simulation: Opportunities and Challenges
• From ChatGPT to DeepSeek: Can LLMs Simulate Humanity?