Human-Curated Data Authoring with LLMs: A Small-Data Approach to Domain Adaptation

2025/06/08 by C. Li, Chance Jiajie Li, Jiayi Wu +28 · 1 voice · 1 citation
Computer Science · Social Sciences · #Language and cultural evolution #Multimodal Machine Learning Applications #Topic Modeling #cs.AI #cs.CY #cs.MA

paper · pdf · doi:10.48550/arxiv.2506.06958

openalex publication_date 2025/06/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

Simulating society with large language models (LLMs), we argue, requires more than generating plausible behavior; it demands cognitively grounded reasoning that is structured, revisable, and traceable. LLM-based agents are increasingly used to emulate individual and group behavior, primarily through prompting and supervised fine-tuning. Yet current simulations remain grounded in a behaviorist "demographics in, behavior out" paradigm, focusing on surface-level plausibility. As a result, they often lack internal coherence, causal reasoning, and belief traceability, making them unreliable for modeling how people reason, deliberate, and respond to interventions. To address this, we present a conceptual modeling paradigm, Generative Minds (GenMinds), which draws from cognitive science to support structured belief representations in generative agents. To evaluate such agents, we introduce the RECAP (REconstructing CAusal Paths) framework, a benchmark designed to assess reasoning fidelity via causal traceability, demographic grounding, and intervention consistency. These contributions advance a broader shift: from surface-level mimicry to generative agents that simulate thought, not just language, for social simulations.

Citations

• Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
• A Roadmap to Pluralistic Alignment
• Large language models that replace human participants can harmfully misportray and flatten identity groups
• From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thought
• Tree of Thoughts: Deliberate Problem Solving with Large Language Models
• Generative Agents: Interactive Simulacra of Human Behavior
• ChatGPT outperforms crowd workers for text-annotation tasks
• A Generalist Agent
• Language Models are Few-Shot Learners

Cited by

• LLM-Based Social Simulations Require a Boundary

Discussions

• "Simulating human society requires more than generating plausible conversations. It requires simulating the structure of human reasoning." Debatable claim, because e.g. it's not clear we have great ac [bsky, 23 points, 4 comments]

Related

• Doing Things with Words: Rethinking Theory of Mind Simulation in Large Language Models
• Integrating LLM in Agent-Based Social Simulation: Opportunities and Challenges
• Large Language Models as Theory of Mind Aware Generative Agents with Counterfactual Reflection
• Mind the (Belief) Gap: Group Identity in the World of LLMs
• Do Large Language Models Solve the Problems of Agent-Based Modeling? A Critical Review of Generative Social Simulations