Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents

2024/11/10 by Yu Gu, 裕二 池谷, Kai Zhang +23 · 3 voices · 62 citations
Computer Science · #Business #Computer science #Digital Rights Management and Security #Mobile Agent-Based Network Management #Multi-Agent Systems and Negotiation #The Internet #World Wide Web

paper · pdf · doi:10.48550/arxiv.2411.06559

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/11/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Language agents based on large language models (LLMs) have demonstrated great promise in automating web-based tasks. Recent work has shown that incorporating advanced planning algorithms, e.g., tree search, is advantageous over reactive planning for web agents. However, unlike simulated sandbox environments, real-world environments such as the web are rife with irreversible actions. This undermines the feasibility of backtracking, a cornerstone of (tree) search. Overly relying on test-time search also hurts efficiency. We advocate model-based planning for web agents that employs a world model to simulate and deliberate over the outcome of each candidate action before committing to one. We systematically explore this paradigm by (1) Proposing a model-based planning framework, WebDreamer, which employs LLMs to serve as both world models and value functions; (2) Training specialized LLMs as world models with a scalable data synthesis pipeline. Empirical results demonstrate that WebDreamer achieves substantial performance improvements over reactive baselines. It is competitive, while being 4-5 times more efficient, with tree search in sandbox environments (VisualWebArena) and also works effectively on real-world websites (Online-Mind2Web and Mind2Web-Live). Furthermore, our trained world model, Dreamer-7B, performs comparable to GPT-4o, highlighting the potential of specialized world models for efficient and effective planning in complex web environments.

Cited by

• The Tool Illusion: Rethinking Tool Use in Web Agents
• Qwen-AgentWorld: Language World Models for General Agents
• Build the web for agents, not agents for the web
• Web World Models
• From Word to World: Can Large Language Models be Implicit Text-based World Models?
• Scaling Agent Learning via Experience Synthesis
• Branch-and-Browse: Efficient and Controllable Web Exploration with Tree-Structured Reasoning and Action Memory
• See, Think, Act: Online Shopper Behavior Simulation with VLM Agents
• Experience-Driven Exploration for Efficient API-Free AI Agents
• PolySkill: Learning Generalizable Skills Through Polymorphic Abstraction
• LLMs as Scalable, General-Purpose Simulators For Evolving Digital Agent Training
• R-WoM: Retrieval-augmented World Model For Computer-use Agents
• Dyna-Mind: Learning to Simulate from Experience for Better AI Agents
• Agent Learning via Early Experience
• Watch and Learn: Learning to Use Computers from Online Videos
• OptAgent: Optimizing Query Rewriting for E-commerce via Multi-Agent Simulation
• WALT: Web Agents that Learn Tools
• EMR-AGENT: Automating Cohort and Feature Extraction from EMR Databases
• Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learning
• UISim: An Interactive Image-Based UI Simulator for Dynamic Mobile Environments
• Recon-Act: A Self-Evolving Multi-Agent Browser-Use System via Web Reconnaissance, Tool Generation, and Task Execution
• Why Are GUI Agents Correct but Late? Decode on the Decision-Time Critical Path, Tested with Pre-Compiled Policy Trees
• TAPO: Transition-Aware Policy Optimization for LLM Agents
• Planning with Reasoning using Vision Language World Model
• Deep Research: A Survey of Autonomous Research Agents
• SSRL: Self-Search Reinforcement Learning
• General Agentic Planning Through Simulative Reasoning with World Models
• OS-MAP: How Far Can Computer-Using Agents Go in Breadth and Depth?
• K-Search: LLM Kernel Generation via Co-Evolving Intrinsic World Model
• WebWorld: A Large-Scale World Model for Web Agent Training
• WebGuard: Building a Generalizable Guardrail for Web Agents
• Toward Real-World Table Agents: Capabilities, Workflows, and Design Principles for LLM-based Table Intelligence
• Role-Agent: Bootstrapping LLM Agents via Dual-Role Evolution
• WebSynthesis: World-Model-Guided MCTS for Efficient WebUI-Trajectory Synthesis
• Pipelined Decoder for Efficient Context-Aware Text Generation
• Universal Retrieval for Multimodal Trajectory Modeling
• Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agents
• Beyond Syntax: Action Semantics Learning for App Agents
• LLMs as World Models: Data-Driven and Human-Centered Pre-Event Simulation for Disaster Impact Assessment
• GOBench: Benchmarking Geometric Optics Generation and Understanding of MLLMs
• Dyna-Think: Synergizing Reasoning, Acting, and World Model Simulation in AI Agents
• WebCoT: Enhancing Web Agent Reasoning by Reconstructing Chain-of-Thought in Reflection, Branching, and Rollback
• Large Language Models for Planning: A Comprehensive and Systematic Survey
• SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Data
• Web-Shepherd: Advancing PRMs for Reinforcing Web Agents
• RLVR-World: Training World Models with Reinforcement Learning
• RMSWeb: Reflection, Failure-Mode Mining, and Salvage-DS for Web Agent Reinforcement Learning
• Quo Vadis, World Modeling?
• Anticipatory Planning for Multimodal AI Agents
• Internalizing the Future: A Unified Agentic Training Paradigm for World Model Planning
• ScaleTrack: Scaling and back-tracking Automated GUI Agents
• Autonomous Continual Learning for Environment Adaptation of Computer-Use Agents
• Generative Visual Code Mobile World Models
• Beyond Clicking:A Step Towards Generalist GUI Grounding via Text Dragging
• Improving Large Language Model Planning with Action Sequence Similarity
• WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Model
• Building LLM Agents by Incorporating Insights from Computer Systems
• WebLists: Extracting Structured Information From Complex Interactive Websites Using Executable LLM Agents
• DreamGuard: Efficient Runtime Guardrail for LLM Agents via Risk-Aware World Model
• AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agents
• REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites
• ViMo: A Generative Visual GUI World Model for App Agents
• SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skills

Discussions

• LLMs might secretly be world models of the internet! By treating LLMs as simulators that can predict "what would happen if I click this?" the authors built an AI that can navigate websites by imagini [bsky, 87 points, 3 comments]
• This paper by Yu Gu et al looks *fascinating* arxiv.org/abs/2411.06559 “Language agents have demonstrated promising capabilities in automating web-based tasks, though their current reactive approache [bsky, 9 points, 1 comments]
• Is Your LLM a World Model of the Internet? Planning for Web Agents [hn, 1 points, 0 comments]

Related

• Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation
• WebATLAS: An LLM Agent with Experience-Driven Memory and Action Simulation
• AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?
• On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark)
• AI Planning Framework for LLM-Based Web Agents