AgentSims: An Open-Source Sandbox for Large Language Model Evaluation

2023/08/08 by Jiaju Lin, Haoran Zhao, Lin, Jiaju +9 · 8 citations
Computer Science · #14J26 (Secondary) 68T42 #14J26 (Secondary) MSC-class: 14J60 (Primary) 14F05 #14J60 (Primary) 14F05 #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2308.04026

openalex publication_date 2023/08/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

With ChatGPT-like large language models (LLM) prevailing in the community, how to evaluate the ability of LLMs is an open question. Existing evaluation methods suffer from following shortcomings: (1) constrained evaluation abilities, (2) vulnerable benchmarks, (3) unobjective metrics. We suggest that task-based evaluation, where LLM agents complete tasks in a simulated environment, is a one-for-all solution to solve above problems. We present AgentSims, an easy-to-use infrastructure for researchers from all disciplines to test the specific capacities they are interested in. Researchers can build their evaluation tasks by adding agents and buildings on an interactive GUI or deploy and test new support mechanisms, i.e. memory, planning and tool-use systems, by a few lines of codes. Our demo is available at https://agentsims.com .

Cited by

• Web World Models
• The Moltbook Files: A Harmless Slopocalypse or Humanity's Last Experiment
• Static Sandboxes Are Inadequate: Modeling Societal Complexity Requires Open-Ended Co-Evolution in LLM-Based Multi-Agent Simulations
• StoryBox: Collaborative Multi-Agent Simulation for Hybrid Bottom-Up Long-Form Story Generation Using Large Language Models
• Simulating Teams with LLM Agents: Interactive 2D Environments for Studying Human-AI Dynamics
• DeepTravel: An End-to-End Agentic Reinforcement Learning Framework for Autonomous Travel Planning Agents
• Chimera: Harnessing Multi-Agent LLMs for Automatic Insider Threat Simulation
• Evaluation and Benchmarking of LLM Agents: A Survey

Related

• AgentBench: Evaluating LLMs as Agents
• AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents
• MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains
• MCPEval: Automatic MCP-based Deep Evaluation for AI Agent Models
• Assessing and Verifying Task Utility in LLM-Powered Applications