Large Language Models as Virtual Survey Respondents: Evaluating Sociodemographic Response Generation

2025/09/08 by Jianpeng Zhao, Zhao, Jianpeng, Chenyu Yuan +15 · 1 voice
Computer Science · Social Sciences · #Artificial Intelligence (cs.AI) #Computational and Text Analysis Methods #FOS: Computer and information sciences #cs.AI

paper · pdf · doi:10.48550/arxiv.2509.06337

openalex publication_date 2025/09/08 · arxiv published 2025/09/08 · openalex created_date 2025/10/11 · arxiv updated 2026/04/27 · openalex updated_date 2026/08/01

Abstract

Questionnaire-based surveys are foundational to social science research and public policymaking, yet traditional survey methods remain costly, time-consuming, and often limited in scale. Although prior work has explored large language models (LLMs) as virtual survey respondents, existing studies often address narrow task settings, focus on single sociological domains, or lack a unified evaluation framework that enables systematic comparison across diverse datasets and models. To address these gaps, we introduce two complementary task abstractions: Partial Attribute Simulation (PAS), where LLMs predict missing attributes from incomplete respondent profiles, and Full Attribute Simulation (FAS), where LLMs generate complete synthetic datasets under zero-context and context-enhanced conditions. Both are framed as diagnostic and exploratory tools rather than replacements for human data collection. We curate LLM-S³ (Large Language Model-based Sociodemographic Survey Simulation), a benchmark spanning 11 real-world public datasets across four sociological domains, and evaluate GPT-3.5/4 Turbo and LLaMA 3.0/3.1-8B under zero-shot and few-shot settings. Our evaluation reveals consistent performance trends across model families, highlights failure modes in structured output generation, and demonstrates how context and prompt design affect simulation fidelity. Our code and dataset are available at: https://github.com/dart-lab-research/LLM-S-Cube-Benchmark

Citations

• On the Generalization vs Fidelity Paradox in Knowledge Distillation
• LLM Social Simulations Are a Promising Research Method
• The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Tasks
• Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations
• ReFF: Reinforcing Format Faithfulness in Language Models across Varied Tasks
• From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agents
• The Llama 3 Herd of Models
• Quantifying and Optimizing Global Faithfulness in Persona-driven Role-playing
• Large Language Models as Instruments of Power: New Regimes of Autonomous Manipulation and Control
• How Well Can LLMs Echo Us? Evaluating AI Chatbots' Role-Play Ability with ECHO
• Human Simulacra: Benchmarking the Personification of Large Language Models
• Random Silicon Sampling: Simulating Human Sub-Population Opinion Using a Large Language Model Based on Group-Level Demographic Information
• Are Large Language Models (LLMs) Good Social Predictors?
• A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications
• PersonalityChat: Conversation Distillation for Personalized Dialog Modeling with Facts and Traits
• InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews
• Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBench
• Do LLMs Possess a Personality? Making the MBTI Test an Amazing Evaluation for Large Language Models
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• AI-Augmented Surveys: Leveraging Large Language Models and Surveys for Opinion Prediction
• LaMP: When Large Language Models Meet Personalization
• GPT-4 Technical Report
• A Survey on In-context Learning
• Out of One, Many: Using Language Models to Simulate Human Samples
• Training language models to follow instructions with human feedback
• Few-Shot Learning with Class Imbalance
• Language Models are Few-Shot Learners
• Scaling Laws for Neural Language Models
• Personalizing Dialogue Agents: I have a dog, do you have pets too?
• Character is Destiny: Can Role-Playing Language Agents Make Persona-Driven Decisions?
• LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals

Cited by

• Can Machines Philosophize?

Discussions

• On est en 2026, y'a une chance que ce ne soit même pas des personnes réelles qui sont sondés. arxiv.org/abs/2509.063... [bsky, 1 points, 0 comments]

Related

• Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations
• SocioBench: Modeling Human Behavior in Sociological Surveys with Large Language Models
• Are Large Language Models Chameleons? An Attempt to Simulate Social Surveys
• Survey Response Generation: Generating Closed-Ended Survey Responses In-Silico with Large Language Models
• When Synthetic Users Fail: A Cross-Domain Benchmark of LLM-Simulated Human Survey Responses