Can Language Models Serve as Text-Based World Simulators?

2024/06/10 by Ruoyao Wang, Graham Todd, Wang, Ruoyao +12 · 2 voices · 17 citations
Computer Science · #Artificial intelligence #Computer science #Multi-Agent Systems and Negotiation #Natural Language Processing Techniques #Natural language processing #cs.AI #cs.CL

paper · pdf · doi:10.48550/arxiv.2406.06485

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/06/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Virtual environments play a key role in benchmarking advances in complex planning and decision-making tasks but are expensive and complicated to build by hand. Can current language models themselves serve as world simulators, correctly predicting how actions change different world states, thus bypassing the need for extensive manual coding? Our goal is to answer this question in the context of text-based simulators. Our approach is to build and use a new benchmark, called ByteSized32-State-Prediction, containing a dataset of text game state transitions and accompanying game tasks. We use this to directly quantify, for the first time, how well LLMs can serve as text-based world simulators. We test GPT-4 on this dataset and find that, despite its impressive performance, it is still an unreliable world simulator without further innovations. This work thus contributes both new insights into current LLM's capabilities and weaknesses, as well as a novel benchmark to track future progress as new models appear.

Cited by

• Instrument Effects in Language-Model Honesty Evaluation: An Auditable Single-System Demonstration
• Qwen-AgentWorld: Language World Models for General Agents
• Evolution and The Knightian Blindspot of Machine Learning
• From Word to World: Can Large Language Models be Implicit Text-based World Models?
• LLM-Driven Corrective Robot Operation Code Generation with Static Text-Based Simulation
• Visual Generation Unlocks Human-Like Reasoning through Multimodal World Models
• R-WoM: Retrieval-augmented World Model For Computer-use Agents
• ByteSized32Refactored: Towards an Extensible Interactive Text Games Corpus for LLM World Modeling and Evaluation
• Foundation Models as World Models: A Foundational Study in Text-Based GridWorlds
• Planning with Reasoning using Vision Language World Model
• G-Sim: Generative Simulations with Large Language Models and Gradient-Free Calibration
• AutoLayout: Closed-Loop Layout Synthesis via Slow-Fast Collaborative Reasoning
• Dyna-Think: Synergizing Reasoning, Acting, and World Model Simulation in AI Agents
• RLVR-World: Training World Models with Reinforcement Learning
• Vid2World: Crafting Video Diffusion Models to Interactive World Models
• Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning
• Evolution of Cooperation in LLM-Agent Societies: A Preliminary Study Using Different Punishment Strategies

Discussions

• Can language models serve as text-based world simulators? [hn, 95 points, 65 comments]
• Can Language Models Serve as Text-Based World Simulators? (answer - NO) arxiv.org/pdf/2406.06485 #chatgpt #llms #artificialintelligence [bsky, 0 points, 0 comments]

Related

• ByteSized32Refactored: Towards an Extensible Interactive Text Games Corpus for LLM World Modeling and Evaluation
• Pre-trained Language Models as Prior Knowledge for Playing Text-based\n Games
• GVGAI-LLM: Evaluating Large Language Model Agents with Infinite Games
• TextWorldExpress: Simulating Text Games at One Million Steps Per Second
• Language-Guided World Models: A Model-Based Approach to AI Control