2025/09/22 by Jiaxu Zhou, Zhou, Jiaxu, Jen-tse Huang +14 · 1 voice · 1 citation
Business, Management and Accounting · Computer Science · #Computation and Language (cs.CL) #Computers and Society (cs.CY) #Corporate Governance and Law #Corporate Insolvency and Governance #FOS: Computer and information sciences #cs.CL #cs.CY
paper · pdf · doi:10.48550/arxiv.2509.18052
openalex publication_date 2025/09/22 · arxiv published 2025/09/22 · openalex created_date 2025/10/10 · arxiv updated 2026/04/06 · openalex updated_date 2026/07/28
Large language models (LLMs) are increasingly deployed to simulate human collective behaviors, yet the methodological rigor of these "AI societies" remains under-explored. Through a systematic audit of 39 recent studies, we identify six pervasive flaws-spanning agent profiles, interaction, memory, control, unawareness, and realism (PIMMUR). Our analysis reveals that 89.7% of studies violate at least one principle, undermining simulation validity. We demonstrate that frontier LLMs correctly identify the underlying social experiment in 50.8% of cases, while 61.0% of prompts exert excessive control that pre-determines outcomes. By reproducing five representative experiments (e.g., telephone game), we show that reported collective phenomena often vanish or reverse when PIMMUR principles are enforced, suggesting that many "emergent" behaviors are methodological artifacts rather than genuine social dynamics. Our findings suggest that current AI simulations may capture model-specific biases rather than universal human social behaviors, raising critical concerns about the use of LLMs as scientific proxies for human society.