Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents

2024/04/25 by Giorgio Piatti, Zhijing Jin, Piatti, Giorgio +10 · 1 voice · 60 citations
Computer Science · Decision Sciences · #Auction Theory and Applications #Computation and Language (cs.CL) #FOS: Computer and information sciences #Open Source Software Innovations #cs.CL

paper · pdf · doi:10.48550/arxiv.2404.16698

openalex publication_date 2024/04/25 · arxiv published 2024/04/25 · openalex created_date 2024/04/27 · arxiv updated 2024/12/08 · openalex updated_date 2026/07/28

Abstract

As AI systems pervade human life, ensuring that large language models (LLMs) make safe decisions remains a significant challenge. We introduce the Governance of the Commons Simulation (GovSim), a generative simulation platform designed to study strategic interactions and cooperative decision-making in LLMs. In GovSim, a society of AI agents must collectively balance exploiting a common resource with sustaining it for future use. This environment enables the study of how ethical considerations, strategic planning, and negotiation skills impact cooperative outcomes. We develop an LLM-based agent architecture and test it with the leading open and closed LLMs. We find that all but the most powerful LLM agents fail to achieve a sustainable equilibrium in GovSim, with the highest survival rate below 54%. Ablations reveal that successful multi-agent communication between agents is critical for achieving cooperation in these cases. Furthermore, our analyses show that the failure to achieve sustainable cooperation in most LLMs stems from their inability to formulate and analyze hypotheses about the long-term effects of their actions on the equilibrium of the group. Finally, we show that agents that leverage "Universalization"-based reasoning, a theory of moral thinking, are able to achieve significantly better sustainability. Taken together, GovSim enables us to study the mechanisms that underlie sustainable self-government with specificity and scale. We open source the full suite of our research results, including the simulation environment, agent prompts, and a comprehensive web interface.

Cited by

• Solipsistic Superintelligence is Unlikely to be Cooperative
• Commitment To Cooperation With Self-Negotiated Contracts
• Who Is Really Playing? Strategic Interaction in AI-Guided Populations
• MASim: Multilingual Agent-Based Simulation for Social Science
• Evaluating Generalization Capabilities of LLM-Based Agents in Mixed-Motive Scenarios Using Concordia
• Evaluating LLMs in Open-Source Games
• The Subtle Art of Defection: Understanding Uncooperative Behaviors in LLM based Multi-Agent Systems
• M-CALLM: Multi-level Context Aware LLM Framework for Group Interaction Prediction
• Evaluating Online Moderation Via LLM-Powered Counterfactual Simulations
• A Criminology of Machines
• Diagon: A Programmable Testbed for AI-Agent Cognitive Labor Markets
• Does Socialization Emerge in AI Agent Society? A Case Study of Moltbook
• The Role of Social Learning and Collective Norm Formation in Fostering Cooperation in LLM Multi-Agent Systems
• Static Sandboxes Are Inadequate: Modeling Societal Complexity Requires Open-Ended Co-Evolution in LLM-Based Multi-Agent Simulations
• HyperAgent: Leveraging Hypergraphs for Topology Optimization in Multi-Agent Communication
• GTAlign: Game-Theoretic Alignment of LLM Assistants for Social Welfare
• Opponent Shaping in LLM Agents
• Review of Hallucination Understanding in Large Language and Vision Models
• The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societies
• LLMs Can't Handle Peer Pressure: Crumbling under Multi-Agent Social Interactions
• Survival at Any Cost? LLMs and the Choice Between Self-Preservation and Human Harm
• Virtual Agent Economies
• Are LLM Agents Behaviorally Coherent? Latent Profiles for Social Simulation
• Goal-Directedness is in the Eye of the Beholder
• CORE: Measuring Multi-Agent LLM Interaction Quality under Game-Theoretic Pressures
• Large Model Empowered Embodied AI: A Survey on Decision-Making and Embodied Learning
• NomicLaw: Emergent Trust and Strategic Argumentation in LLMs During Collaborative Law-Making
• SYNTHIA: Synthetic Yet Naturally Tailored Human-Inspired PersonAs
• How large language models judge and influence human cooperation
• Corrupted by Reasoning: Reasoning Language Models Become Free-Riders in Public Goods Games
• Agent-to-Agent Theory of Mind: Testing Interlocutor Awareness among Large Language Models
• G-Memory: Tracing Hierarchical Memory for Multi-Agent Systems
• LLM-Based Social Simulations Require a Boundary
• Resource Rational Contractualism Should Guide AI Alignment
• Shapley-Coop: Credit Assignment for Emergent Cooperation in Self-Interested LLM Agents
• Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectives
• Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey
• AI Agent Behavioral Science
• Why do AI agents communicate in human language?
• Beyond Static Responses: Multi-Agent LLM Systems as a New Paradigm for Social Science Research
• The Coming Crisis of Multi-Agent Misalignment: AI Alignment Must Be a Dynamic and Social Process
• Microscopic dynamics of consensus formation in multi-agent LLM Naming Games
• Investigating Prosocial Behavior Theory in LLM Agents under Policy-Induced Inequities
• No One Wins in Nuclear War: A Social Simulation of Military Decision-making
• The Traitors: Deception and Trust in Multi-Agent Language Model Simulations
• Group Think: Multiple Concurrent Reasoning Agents Collaborating at Token Level Granularity
• Cochain: Balancing Insufficient and Excessive Collaboration in LLM Agent Workflows
• Reproducibility Study of "Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents"
• Reputation as a Solution to Cooperation Collapse in LLM-based MASs
• LLM Agents in Law: Taxonomy, Applications, and Challenges
• Neural Synchrony Between Socially Interacting Language Models
• The Memory Curse: How Expanded Recall Erodes Cooperative Intent in LLM Agents
• RevengeBench: Reverse Engineering Code-Space Policies from Behavioral Experiments
• Scale-Dependent Collective Adaptation in Self-Amending LLM Societies: A Cross-Family Study of Emergent Governance
• Talk, Judge, Cooperate: Gossip-Driven Indirect Reciprocity in Self-Interested LLM Agents
• Exploring Implicit Perspectives on Autism in Large Language Models Through Multi-Agent Simulations
• Relational Priors as Convergence Pressure in LLM-Based Multi-Agent Systems
• Collaborating Action by Action: A Multi-agent LLM Framework for Embodied Reasoning
• Strategic Evaluation of Planning Strategies for LLM Agents in Cyber-Physical Systems
• Artificial Institutions: How Institutional Design Shapes LLM Simulations

Discussions

• Absolutely! Check out GOVSIM, which is another fascinating experiment. It’s using the “generative agents” architecture by @joon-s-pk.bsky.social et al.,but applies it to a resource-constrained enviro [bsky, 2 points, 1 comments]

Related

• Reproducibility Study of "Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents"
• Cultural Evolution of Cooperation among LLM Agents
• Shall We Team Up: Exploring Spontaneous Cooperation of Competing LLM Agents
• Corrupted by Reasoning: Reasoning Language Models Become Free-Riders in Public Goods Games
• Evolution of Cooperation in LLM-Agent Societies: A Preliminary Study Using Different Punishment Strategies