Evaluating the Use of Large Language Models as Synthetic Social Agents in Social Science Research

2025/09/30 by Madden, Emma Rose
#Applications (stat.AP) #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2509.26080

Abstract

Large Language Models (LLMs) are being increasingly used as synthetic agents in social science, in applications ranging from augmenting survey responses to powering multi-agent simulations. This paper outlines cautions that should be taken when interpreting LLM outputs and proposes a pragmatic reframing for the social sciences in which LLMs are used as high-capacity pattern matchers for quasi-predictive interpolation under explicit scope conditions and not as substitutes for probabilistic inference. Practical guardrails such as independent draws, preregistered human baselines, reliability-aware validation, and subgroup calibration, are introduced so that researchers may engage in useful prototyping and forecasting while avoiding category errors.

Citations

• Predicting Field Experiments with Large Language Models
• LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals
• Self-Preference Bias in LLM-as-a-Judge
• From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
• Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge
• Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspective
• Synthetic Replacements for Human Survey Data? The Perils of Large Language Models
• Humans or LLMs as the Judge? A Study on Judgement Biases
• In-Context Language Learning: Architectures and Algorithms
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• In-Context Learning through the Bayesian Prism
• Turning large language models into cognitive models
• Generative Agents: Interactive Simulacra of Human Behavior
• Whose Opinions Do Language Models Reflect?
• Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?
• Out of One, Many: Using Language Models to Simulate Human Samples
• Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies
• Emergent Abilities of Large Language Models
• An Explanation of In-context Learning as Implicit Bayesian Inference
• On the Opportunities and Risks of Foundation Models
• Language Models are Few-Shot Learners
• Scaling Laws for Neural Language Models
• Attention Is All You Need
• Limit theorems for a class of identically distributed random variables
• How Many Human Survey Respondents is a Large Language Model Worth? An Uncertainty Quantification Perspective

Related

• The Emergence of Social Science of Large Language Models
• Selecting Language Models for Social Science: Start Small, Start Open, and Validate
• Validating LLMs in social science: Epistemic threats and emerging norms
• Beyond Static Responses: Multi-Agent LLM Systems as a New Paradigm for Social Science Research
• Large Language Models in Human Subject Research, and the Presence of Idiosyncratic Human Behaviors