Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents

2024/11/10 by Yu Gu, 裕二 池谷, Kai Zhang +23 · 3 voices · 35 citations
Computer Science · #Multi-Agent Systems and Negotiation #Digital Rights Management and Security #Mobile Agent-Based Network Management

paper · pdf · doi:10.48550/arxiv.2411.06559

Abstract

Language agents based on large language models (LLMs) have demonstrated great promise in automating web-based tasks. Recent work has shown that incorporating advanced planning algorithms, e.g., tree search, is advantageous over reactive planning for web agents. However, unlike simulated sandbox environments, real-world environments such as the web are rife with irreversible actions. This undermines the feasibility of backtracking, a cornerstone of (tree) search. Overly relying on test-time search also hurts efficiency. We advocate model-based planning for web agents that employs a world model to simulate and deliberate over the outcome of each candidate action before committing to one. We systematically explore this paradigm by (1) Proposing a model-based planning framework, WebDreamer, which employs LLMs to serve as both world models and value functions; (2) Training specialized LLMs as world models with a scalable data synthesis pipeline. Empirical results demonstrate that WebDreamer achieves substantial performance improvements over reactive baselines. It is competitive, while being 4-5 times more efficient, with tree search in sandbox environments (VisualWebArena) and also works effectively on real-world websites (Online-Mind2Web and Mind2Web-Live). Furthermore, our trained world model, Dreamer-7B, performs comparable to GPT-4o, highlighting the potential of specialized world models for efficient and effective planning in complex web environments.

Cited by

• The Tool Illusion: Rethinking Tool Use in Web Agents
• Qwen-AgentWorld: Language World Models for General Agents
• Build the web for agents, not agents for the web
• Web World Models
• From Word to World: Can Large Language Models be Implicit Text-based World Models?
• Scaling Agent Learning via Experience Synthesis
• Branch-and-Browse: Efficient and Controllable Web Exploration with Tree-Structured Reasoning and Action Memory
• See, Think, Act: Online Shopper Behavior Simulation with VLM Agents
• Experience-Driven Exploration for Efficient API-Free AI Agents
• PolySkill: Learning Generalizable Skills Through Polymorphic Abstraction
• LLMs as Scalable, General-Purpose Simulators For Evolving Digital Agent Training
• R-WoM: Retrieval-augmented World Model For Computer-use Agents
• Dyna-Mind: Learning to Simulate from Experience for Better AI Agents
• Agent Learning via Early Experience
• Watch and Learn: Learning to Use Computers from Online Videos
• OptAgent: Optimizing Query Rewriting for E-commerce via Multi-Agent Simulation
• WALT: Web Agents that Learn Tools
• EMR-AGENT: Automating Cohort and Feature Extraction from EMR Databases
• Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learning
• UISim: An Interactive Image-Based UI Simulator for Dynamic Mobile Environments
• Recon-Act: A Self-Evolving Multi-Agent Browser-Use System via Web Reconnaissance, Tool Generation, and Task Execution
• Why Are GUI Agents Correct but Late? Decode on the Decision-Time Critical Path, Tested with Pre-Compiled Policy Trees
• TAPO: Transition-Aware Policy Optimization for LLM Agents
• Planning with Reasoning using Vision Language World Model
• Deep Research: A Survey of Autonomous Research Agents
• SSRL: Self-Search Reinforcement Learning
• General Agentic Planning Through Simulative Reasoning with World Models
• OS-MAP: How Far Can Computer-Using Agents Go in Breadth and Depth?
• K-Search: LLM Kernel Generation via Co-Evolving Intrinsic World Model
• WebWorld: A Large-Scale World Model for Web Agent Training
• WebGuard: Building a Generalizable Guardrail for Web Agents
• Toward Real-World Table Agents: Capabilities, Workflows, and Design Principles for LLM-based Table Intelligence
• WebSynthesis: World-Model-Guided MCTS for Efficient WebUI-Trajectory Synthesis
• Pipelined Decoder for Efficient Context-Aware Text Generation
• Universal Retrieval for Multimodal Trajectory Modeling

Discussions

• LLMs might secretly be world models of the internet! By treating LLMs as simulators that can predict "what would happen if I click this?" the authors built an AI that can navigate websites by imagini [bsky, 87 points, 3 comments]
• This paper by Yu Gu et al looks *fascinating* arxiv.org/abs/2411.06559 “Language agents have demonstrated promising capabilities in automating web-based tasks, though their current reactive approache [bsky, 9 points, 1 comments]
• Is Your LLM a World Model of the Internet? Planning for Web Agents [hn, 1 points, 0 comments]

Related

• Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation
• WebATLAS: An LLM Agent with Experience-Driven Memory and Action Simulation
• AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?
• On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark)
• On the Planning Abilities of Large Language Models : A Critical Investigation