LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals

2024/11/15 by Joon Sung Park, Park, Joon Sung, Carolyn Q. Zou +20 · 64 voices · 94 citations
Mathematics · Decision Sciences · #COVID-19 epidemiological studies #Innovation Diffusion and Forecasting

paper · pdf · doi:10.48550/arxiv.2411.10109

Abstract

Machine learning can predict human behavior well when substantial structured data are available for well-defined outcomes. Such models are typically outcome-specific, however, requiring training data for each target outcome, limiting their applicability to new domains. We test whether large language models (LLMs) can relax these requirements by using self-report data to build attitudinal and behavioral simulations, or "generative agents," that can predict responses across outcomes without outcome-specific training data. Using data from a diverse national sample of 1,052 Americans, we built agents from (i) two-hour, semi-structured interviews elicited using the American Voices Project interview schedule, (ii) structured surveys including General Social Survey items and the Big Five personality inventory, or (iii) both sources combined. On held-out General Social Survey items, interview-only, survey-only, and combined agents achieved accuracies equal to 83%, 82%, and 86% of participants' own two-week test-retest consistency benchmark, respectively, compared with 74% for demographics-only agents. Combining interviews and surveys produced the highest accuracy, though gains over either source alone were modest, suggesting that predictive benefits from data begin to asymptote once the model has observed sufficient evidence within a domain. We find that these agents also predict personality traits, economic-game behavior, and experimental responses, while reducing accuracy disparities across racial and ideological groups relative to demographics-only agents. Together, these results show that LLM agents grounded in qualitative or quantitative self-reports can support general-purpose simulation of individuals across outcomes, without requiring task-specific training data.

Cited by

• After Talking with 1,000 Personas: Learning Preference-Aligned Proactive Assistants From Large-Scale Persona Interactions
• Computational Turing Test Reveals Systematic Differences Between Human and AI Language
• A Pragmatic View of AI Personhood
• "My Boyfriend is AI": A Computational Analysis of Human-AI Companionship in Reddit's AI Community
• Large Language Models Do Not Simulate Human Psychology
• Technological folie à deux: Feedback Loops Between AI Chatbots and Mental Illness
• LLM Social Simulations Are a Promising Research Method
• Value Profiles for Encoding Human Variation
• Computational structuralism: Toward a formal theory of meaning in the age of digital intelligence
• Informing AI Policy Assessment using Large-Scale Simulation of Interventions
• Instruction-Tuned Language Models Cannot Sample from Distributions They Can Describe
• Quantifying the Lifelong Impact of Resilience Interventions via Agent-Based LLM Simulation
• Emergent Bias and Fairness in Multi-Agent Decision Systems
• Generative AI as Digital Representatives in Collective Decision-Making: A Game-Theoretical Approach
• CP-Env: Evaluating Large Language Models on Clinical Pathways in a Controllable Hospital Environment
• MASim: Multilingual Agent-Based Simulation for Social Science
• Measuring Agents in Production
• A Meta-Analysis of the Persuasive Power of Large Language Models
• Mitigating the Threshold Priming Effect in Large Language Model-Based Relevance Judgments via Personality Infusing
• Misalignment of LLM-Generated Personas with Human Perceptions in Low-Resource Settings
• The Subtle Art of Defection: Understanding Uncooperative Behaviors in LLM based Multi-Agent Systems
• German General Social Survey Personas: A Survey-Derived Persona Prompt Collection for Population-Aligned LLM Studies
• Two-Faced Social Agents: Context Collapse in Role-Conditioned Large Language Models
• Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors
• DeepPersona: A Generative Engine for Scaling Deep Synthetic Personas
• From Five Dimensions to Many: Large Language Models as Precise and Interpretable Psychological Profilers
• Do Androids Dream of Unseen Puppeteers? Probing for a Conspiracy Mindset in Large Language Models
• Rethinking LLM Human Simulation: When a Graph is What You Need
• Simulating and Experimenting with Social Media Mobilization Using LLM Agents
• A Persona-based Rate Action Index
• Will Scaling Improve Social Simulation with LLMs?
• Human diversity fuels collective creativity that large language models cannot simulate or sustain
• JudgeMeNot: Personalizing Large Language Models to Emulate Judicial Reasoning in Hebrew
• How Well Can LLM Agents Simulate End-User Security and Privacy Attitudes and Behaviors?
• Persona Generators: Generating Diverse Synthetic Personas for Arbitrary Contexts
• The Sign Estimator: LLM Alignment in the Face of Choice Heterogeneity
• Group size effects and collective misalignment in LLM multi-agent systems
• Measure what Matters: Psychometric Evaluation of AI with Situational Judgment Tests
• See, Think, Act: Online Shopper Behavior Simulation with VLM Agents
• SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors
• Who's Asking? Simulating Role-Based Questions for Conversational AI Evaluation
• DPRF: A Generalizable Dynamic Persona Refinement Framework for Optimizing Behavior Alignment Between Personalized LLM Role-Playing Agents and Humans
• Static Sandboxes Are Inadequate: Modeling Societal Complexity Requires Open-Ended Co-Evolution in LLM-Based Multi-Agent Simulations
• Missing the Margins: A Systematic Literature Review on the Demographic Representativeness of LLMs
• Addressing the alignment problem in transportation policy making: an LLM approach
• Investigating Political and Demographic Associations in Large Language Models Through Moral Foundations Theory
• From Delegates to Trustees: How Optimizing for Long-Term Interests Shapes Bias and Alignment in LLM
• Too Open for Opinion? Embracing Open-Endedness in Large Language Models for Social Simulation
• Survey Response Generation: Generating Closed-Ended Survey Responses In-Silico with Large Language Models
• Learning to Make MISTAKEs: Modeling Incorrect Student Thinking And Key Errors
• SusBench: An Online Benchmark for Evaluating Dark Pattern Susceptibility of Computer-Use Agents
• To Mask or to Mirror: Human-AI Alignment in Collective Reasoning
• Student Development Agent: Risk-free Simulation for Evaluating AIED Innovations
• Customer-R1: Personalized Simulation of Human Behaviors via RL-based LLM Agent in Online Shopping
• Evaluating LLM Safety Across Child Development Stages: A Simulated Agent Approach
• PrimeX: A Dataset of Worldview, Opinion, and Explanation
• Evaluating the Use of Large Language Models as Synthetic Social Agents in Social Science Research
• MADS: Multi-Agent Dialogue Simulation for Diverse Persuasion Data Generation
• When Can AI Models Explain Learning? Validity Criteria for AI as Cognitive Models in Education
• What Makes LLM Agent Simulations Useful for Policy? Insights From an Iterative Design Engagement in Emergency Preparedness
• Few-Shot and Training-Free Review Generation via Conversational Prompting
• This human study did not involve human subjects: Validating LLM simulations as behavioral evidence
• A large-scale evaluation of commonsense knowledge in humans and large language models
• Simulating Online Social Media Conversations on Controversial Topics Using AI Agents Calibrated on Real-World Data
• Through the Lens of Human-Human Collaboration: A Configurable Research Platform for Exploring Human-Agent Collaboration
• The Narcissus Hypothesis: Descending to the Rung of Illusion
• Predicting Language Models' Success at Zero-Shot Probabilistic Prediction
• Charting trajectories of human thought using large language models
• PREFINE: Personalized Story Generation via Simulated User Critics and User-Specific Rubric Generation
• A Visualized Framework for Event Cooperation with Generative Agents
• MillStone: How Open-Minded Are LLMs?
• Evalet: Evaluating Large Language Models through Functional Fragmentation
• When Your Boss Is an AI Bot: Exploring Opportunities and Risks of Manager Clone Agents in the Future Workplace
• RecoWorld: Building Simulated Environments for Agentic Recommender Systems
• Ethical Frameworks for Conducting Social Challenge Studies
• Talking to an AI Mirror: Designing Self-Clone Chatbots for Enhanced Engagement in Digital Mental Health Support
• Large Language Models as Virtual Survey Respondents: Evaluating Sociodemographic Response Generation
• Finetuning LLMs for Human Behavior Prediction in Social Science Experiments
• Are LLM Agents Behaviorally Coherent? Latent Profiles for Social Simulation
• PersonaTeaming: Exploring How Introducing Personas Can Improve Automated AI Red-Teaming
• Batch Query Processing and Optimization for Agentic Workflows
• ChatCLIDS: Simulating Persuasive AI Dialogues to Promote Closed-Loop Insulin Adoption in Type 1 Diabetes Care
• ChatThero: An LLM-Supported Chatbot for Behavior Change and Therapeutic Support in Addiction Recovery
• Not There Yet: Evaluating Vision Language Models in Simulating the Visual Perception of People with Low Vision
• Simulating Generative Social Agents via Theory-Informed Workflow Design
• IROTE: Human-like Traits Elicitation of Large Language Model via In-Context Self-Reflective Optimization
• Scaling Personality Control in LLMs with Big Five Scaler Prompts
• VirtLab: An AI-Powered System for Flexible, Customizable, and Large-scale Team Simulations
• Pay What LLM Wants: Can LLM Simulate Economics Experiment with 522 Real-human Persona?
• Social Media Information Operations
• Polymorphic Combinatorial Frameworks (PCF): Guiding the Design of Mathematically-Grounded, Adaptive AI Agents
• Accessibility Scout: Personalized Accessibility Scans of Built Environments
• PersonaTwin: A Multi-Tier Prompt Conditioning Framework for Generating and Evaluating Personalized Digital Twins
• Validating Generative Agent-Based Models of Social Norm Enforcement: From Replication to Novel Predictions

Discussions

• Simulating human behavior with AI agents promises a testbed for policy and the social sciences. We interviewed 1,000 people for two hours each to create generative agents of them. These agents replica [bsky, 111 points, 8 comments]
• I am genuinely curious abt the planned uses for these agents Science is asking questions & allowing for unexpected answers How can an agent surprise us in an experiment? In a survey? An agent is a [bsky, 57 points, 10 comments]
• AI agents, suitably prompted, provide responses to GSS that are about 85% as accurate as the respondents themselves arxiv.org/abs/2411.10109 [bsky, 32 points, 3 comments]
• this is absolutely wild arxiv.org/abs/2411.10109 [bsky, 31 points, 3 comments]
• This paper sparked lively discussion in our seminar. Lots of excitement mixed with valid concerns. arxiv.org/abs/2411.10109 [bsky, 20 points, 0 comments]
• This paper transforms 2hr interviews into psychological scores. I have a lot of “psychologist feelings” about this, but it does highlight how a language model is just y = f(x) where x is language and [bsky, 8 points, 1 comments]
• Nedaudz pašapziņu graujošs pētījums par to, cik viegli AI ir replicēt cilvēka personību pēc 2h intervijas (atbildēs pareizi uz 85% no attieksmju jautājumiem). No vienas puses, labi - kļūs par ērtu a [bsky, 5 points, 1 comments]
• Researchers have created 1000 AI ‘agents’ that replicate the responses (in surveys and lab studies) of a representative sample of 1000 real Americans 🤯 arxiv.org/pdf/2411.10109 [bsky, 5 points, 3 comments]
• Did they have 85% accuracy in predicting behavioral questions on a sample of human participants though? :) arxiv.org/abs/2411.10109 [bsky, 5 points, 0 comments]
• Generative Agent Simulations of 1k People [hn, 4 points, 1 comments]
• Some researchers are currently trying to replace human interviewees in the social sciences with large language models. An example is this article by Stanford/DeepMind. arxiv.org/abs/2411.10109 [bsky, 4 points, 1 comments]
• Un estudi ha plasmat la personalitat de 1.052 persones en models d'IA. Per a generar els models s'han fet servir les dades extretes a partir d'entrevistes i enquestes, on els subjectes tractàvem desde [bsky, 4 points, 2 comments]
• arxiv.org/abs/2411.10109 Maybe we could do cool social science research (or propaganda and micromarketing shhh) if we had AIs that predict individual humans' behavior well. Based on qualitative interv [bsky, 3 points, 0 comments]
• 10/ "The Singularity won’t arrive with killer robots. It’s here, hidden in 1,000 detailed interviews, quietly replicating the human experience. 🤯" Paper: arxiv.org/abs/2411.10109 [bsky, 3 points, 0 comments]
• It could be this one? Generative Agent Simulations of 1,000 People arxiv.org/abs/2411.10109 [bsky, 3 points, 1 comments]
• Unless arxiv.org/abs/2411.10109 (not at all claiming this is useful nor that the individuals are being generated btw. just funny how 1000 popped up) [bsky, 3 points, 1 comments]
• Generative Agent Simulations of 1k People [hn, 2 points, 0 comments]
• 🧪 🩺🖥️ #MLSky Direct link to the pre-print arxiv.org/pdf/2411.10109 [bsky, 2 points, 0 comments]
• Even without fine-tuning, GPT-4o can do a fairly decent job of modeling personalities. With fine-tuning, we are very much at the point of being able to create a virtual representative for a group. [bsky, 2 points, 1 comments]
• Generative Agent Simulations of 1k People [pdf] [hn, 2 points, 1 comments]
• Research Paper: Generative Agent Simulations of 1k People [hn, 2 points, 0 comments]
• This might be the wildest paper I have seen in 2024. They simulated 1000 REAL personalities from study subjects by using 2-hour long-form interviews. We are living in a Black Mirror episode. arxiv.org [bsky, 2 points, 0 comments]
• Interview 1000 random people for 2 hours each, and train LLMs on them. The generative agents reflect respondents' personality and views, and mimic the results of population surveys. arxiv.org/abs/241 [bsky, 2 points, 0 comments]
• arxiv.org/abs/2411.10109 [bsky, 2 points, 0 comments]
• Yes, they did. arxiv.org/abs/2411.10109 [bsky, 2 points, 0 comments]
• This arxiv paper is literally the plot of A Mind Forever Voyaging arxiv.org/abs/2411.10109 [bsky, 2 points, 0 comments]
• I think this depends on how the personality is induced. I wouldn't expect much validity from a plain LLM with just a persona but my expectation would be different with fine tuning or just more empiric [bsky, 1 points, 1 comments]
• I often say:"humans need to work harder to do things that only humans can do". Yet AI is ever more powerful. What do we need to infuse into our models of education to elevate human capacities and comp [bsky, 1 points, 0 comments]
• It’s in the study. I didn’t see the questionnaire itself but you can see the interview with the participants starting on page 20. On page 61 you can see all the interview questions arxiv.org/pdf/241 [bsky, 1 points, 0 comments]
• *If I'm reading this correctly, there ought to be some way to tune your Slop You into a Black Slop You and then see what *he* thinks arxiv.org/pdf/2411.10109 [bsky, 1 points, 0 comments]
• Generative Agent Simulations of 1,000 People arxiv.org/abs/2411.10109 [bsky, 1 points, 0 comments]
• Generative Agent Simulations: We've seen this one before with replicated "human-like qualities" in individual "sprites" using an LLM in an interactive silo: arxiv.org/abs/2411.10109 -- but this one is [bsky, 1 points, 0 comments]
• Generative Agent Simulations of 1k People [hn, 1 points, 0 comments]
• Generative Agent Simulations of 1k People [hn, 1 points, 0 comments]
• Generative Agent Simulations of 1k People [hn, 1 points, 0 comments]
• Generative Agent Simulations of 1k People [hn, 1 points, 0 comments]
• Important caveat to the hype: At no point did a *human being* actually interview anyone. As detailed in their research paper, an *AI* interviewer was used and the agent simulations were then compared [bsky, 1 points, 1 comments]
• Read the full paper here: arxiv.org/pdf/2411.10109 h/t @mmitchell_ai for the link [bsky, 1 points, 0 comments]
• Second: Generative Agent Simulations of 1,000 People (arxiv.org/abs/2411.10109) uses long qualitative interviews to simulate personas with very high correlation with traditional survey results. [bsky, 1 points, 0 comments]
• The next AI capability is not coding — it is simulating people [lemmy, 1 points, 0 comments]
• A new project launched! Generative Agent Simulations of 1,000 People. Stanford + DeepMind study that used LLMs to simulate people with AI agents based on a 2-hour interview Check it out: arxiv.org/pdf [bsky, 1 points, 0 comments]
• Generative Agent Simulations of 1,000 People #news #worldnews [bsky, 0 points, 0 comments]
• this seems to be working? not sure what the issue was. arxiv.org/abs/2411.10109 [bsky, 0 points, 1 comments]
• This is an unheard of degree of techno-cynicism. I've seen a lot over the past 30 years, but this is a totally new level. These... 'researchers' used interviews of real people to instruct a model how [bsky, 0 points, 0 comments]
• #GenerativeAI #SocialScience #HumanBehavior #AIApplications #EquityInAI [bsky, 0 points, 0 comments]
• Recent research from Stanford University and Google DeepMind reveals that AI can replicate individual personalities with 85% accuracy after just a two-hour interview. Research paper: arxiv.org/pdf/2 [bsky, 0 points, 0 comments]
• https://bsky.app/profile/cyclicircuit.bsky.social/post/3lf52ja5rt22v [bsky, 0 points, 0 comments]
• 2411.10109] Generative Agent Simulations of 1,000 People [https://arxiv.org/abs/2411.10109 [bsky, 0 points, 0 comments]
• An important paper. arxiv.org/abs/2411.10109 [bsky, 0 points, 0 comments]
• „The generative agents replicate participants' responses on the General Social Survey 85% as accurately as participants replicate their own answers two weeks later, and perform comparably in predictin [bsky, 0 points, 0 comments]
• Unmentioned in the article - 1) the INTERVIEWER was an LLM; 2) they had to keep 'reminding' the model what their previous estimate vis a vis imitating the person's interviews was arxiv.org/pdf/2411.10 [bsky, 0 points, 0 comments]
• @mauricioricardo.bsky.social arxiv.org/abs/2411.101... [bsky, 0 points, 0 comments]
• AI agents for social science simulations -- arxiv.org/abs/2411.10109 [bsky, 0 points, 0 comments]
• arxiv.org/abs/2411.10109 [bsky, 0 points, 1 comments]
• Yes AI can Clone Your Personality in 2 hours arxiv.org/pdf/2411.10109 [bsky, 0 points, 0 comments]
• A study led by researchers at Stanford University has discovered that all it takes is a two-hour interview for an AI model to predict people’s responses to a battery of questionnaires, personality te [bsky, 0 points, 0 comments]
• Society digital twins? Generative Agent Simulations of 1,000 People arxiv.org/pdf/2411.10109 [bsky, 0 points, 1 comments]
• If AI agents can mimic decision making of various humans, this opens up a huge possibility to accurately model society in a large simulation. arxiv.org/abs/2411.10109 [bsky, 0 points, 0 comments]
• Connaissez-vous la psychohistoire ? 📓 Imaginée par Isaac Asimov dans sa série Fondation, la psychohistoire est une discipline fictive combinant mathématiques, psychologie et statistiques pour prédi [bsky, 0 points, 1 comments]
• https://bsky.app/profile/nprtheperson.bsky.social/post/3lbxhdvsfcs2d [bsky, 0 points, 0 comments]
• Link: arxiv.org/pdf/2411.10109 [bsky, 0 points, 0 comments]
• 1000人のリアル人間のインタビュー結果を使って疑似思考AIを作って政策などの反応を予測するとな arxiv.org/abs/2411.10109 [bsky, 0 points, 0 comments]
• 🤖 Stanford researchers used GPT-4o to create AI clones that mimic human personalities with up to 85% accuracy after just a 2-hour interview! 🚀 Potential in social science, policymaking, and custom [bsky, 0 points, 0 comments]
• ”The generative agents replicate participants' responses on the General Social Survey 85% as accurately as participants replicate their own answers two weeks later, and perform comparably in predictin [bsky, 0 points, 0 comments]

Related

• Are LLM Agents Behaviorally Coherent? Latent Profiles for Social Simulation
• Evaluating the Simulation of Human Personality-Driven Susceptibility to Misinformation with LLMs
• Limited Ability of LLMs to Simulate Human Psychological Behaviours: a Psychometric Analysis
• Can Finetuing LLMs on Small Human Samples Increase Heterogeneity, Alignment, and Belief-Action Coherence?
• LLM-Mirror: A Generated-Persona Approach for Survey Pre-Testing