WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learning

2025/05/22 by Zhepei Wei, Wenlin Yao, Wei, Zhepei +21 · 55 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2505.16421

openalex publication_date 2025/05/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

While reinforcement learning (RL) has demonstrated remarkable success in enhancing large language models (LLMs), it has primarily focused on single-turn tasks such as solving math problems. Training effective web agents for multi-turn interactions remains challenging due to the complexity of long-horizon decision-making across dynamic web interfaces. In this work, we present WebAgent-R1, a simple yet effective end-to-end multi-turn RL framework for training web agents. It learns directly from online interactions with web environments by asynchronously generating diverse trajectories, entirely guided by binary rewards depending on task success. Experiments on the WebArena-Lite benchmark demonstrate the effectiveness of WebAgent-R1, boosting the task success rate of Qwen-2.5-3B from 6.1% to 33.9% and Llama-3.1-8B from 8.5% to 44.8%, significantly outperforming existing state-of-the-art methods and strong proprietary models such as OpenAI o3. In-depth analyses reveal the effectiveness of the thinking-based prompting strategy and test-time scaling through increased interactions for web tasks. We further investigate different RL initialization policies by introducing two variants, namely WebAgent-R1-Zero and WebAgent-R1-CoT, which highlight the importance of the warm-up training stage (i.e., behavior cloning) and provide insights on incorporating long chain-of-thought (CoT) reasoning in web agents.

Citations

• Beyond Outcome Reward: Decoupling Search and Answering Improves LLM Agents
• Spurious Rewards: Rethinking Training Signals in RLVR
• The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning
• RAST: Reasoning Activation in LLMs via Small-model Transfer
• ZeroSearch: Incentivize the Search Capability of LLMs without Searching
• RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning
• SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild
• SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks
• Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
• SelfElicit: Your Language Model Secretly Knows Where is the Relevant Evidence
• WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning
• OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization
• Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation
• WebPilot: A Versatile and Autonomous Multi-Agent System for Web Task Execution with Strategic Exploration
• Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents
• VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents
• Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systems
• DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning
• Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization
• Large Language Models Can Self-Improve At Web Agent Tasks
• SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
• Autonomous Evaluation and Refinement of Digital Agents
• WILBUR: Adaptive In-Context Learning for Robust and Accurate Web Agents
• AutoWebGLM: A Large Language Model-based Web Navigating Agent
• AutoGuide: Automated Generation and Selection of Context-Aware Guidelines for Large Language Model Agents
• ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL
• Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Models
• Dual-View Visual Contextualization for Web Navigation
• GPT-4V(ision) is a Generalist Web Agent, if Grounded
• Agent Lumos: Unified and Modular Training for Open-Source Language Agents
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• SteP: Stacked LLM Policies for Web Actions
• LASER: LLM Agent with State-Space Exploration for Web Navigation
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Autonomous Tester Agent Benchmark
• A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
• Mind2Web: Towards a Generalist Agent for the Web
• Take a Break in the Middle: Investigating Subgoals towards Hierarchical Script Generation
• CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society
• Language Models of Code are Few-Shot Commonsense Learners
• Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding
• WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
• MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge
• Show Me More Details: Discovering Hierarchies of Procedures from Semi-structured Web Data
• Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents
• AndroidEnv: A Reinforcement Learning Platform for Android
• Mapping Natural Language Instructions to Mobile UI Action Sequences
• ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• VirtualHome: Simulating Household Activities via Programs
• Proximal Policy Optimization Algorithms
• High-Dimensional Continuous Control Using Generalized Advantage\n Estimation
• VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks

Cited by

• From Word to World: Can Large Language Models be Implicit Text-based World Models?
• ESearch-R1: Learning Cost-Aware MLLM Agents for Interactive Embodied Search via Reinforcement Learning
• DraCo: Draft as CoT for Text-to-Image Preview and Rare Concept Generation
• GTM: Simulating the World of Tools for AI Agents
• Training High-Level Schedulers with Execution-Feedback Reinforcement Learning for Long-Horizon GUI Automation
• OpenApps: Simulating Environment Variations to Measure UI-Agent Reliability
• Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization
• Adapting Web Agents with Synthetic Supervision
• Scaling Agent Learning via Experience Synthesis
• Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberation
• HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Models
• MICA: Multi-granularity Intertemporal Credit Assignment for Long-Horizon Emotional Support Dialogue
• Optimizing Retrieval for RAG via Reinforced Contrastive Learning
• Rank-GRPO: Training LLM-based Conversational Recommender Systems with Reinforcement Learning
• Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domains
• A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications
• WEBSERV: A Full-Stack and RL-Ready Web Environment for Training Web Agents at Scale
• Towards Agentic Self-Learning LLMs in Search Environment
• DeepPlanner: Scaling Planning Capability for Deep Research Agents via Advantage Shaping
• A Survey on Agentic Multimodal Large Language Models
• MTSQL-R1: Towards Long-Horizon Multi-Turn Text-to-SQL via Agentic Training
• Autonomous Agents for Scientific Discovery: Orchestrating Scientists, Language, Code, and Physics
• Dyna-Mind: Learning to Simulate from Experience for Better AI Agents
• Agent Learning via Early Experience
• Customer-R1: Personalized Simulation of Human Behaviors via RL-based LLM Agent in Online Shopping
• Beyond Outcome Reward: Decoupling Search and Answering Improves LLM Agents
• JEF-Hinter: Leveraging Offline Knowledge for Improving Web Agents Adaptation
• WebGen-Agent: Enhancing Interactive Website Generation with Multi-Level Feedback and Step-Level Reinforcement Learning
• Agentic Reinforcement Learning with Implicit Step Rewards
• ARE: Scaling Up Agent Environments and Evaluations
• TGPO: Tree-Guided Preference Optimization for Robust Web Agent Reinforcement Learning
• Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agents
• SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agents
• Symbolic Graphics Programming with Large Language Models
• EviNote-RAG: Enhancing RAG Models via Answer-Supportive Evidence Notes
• UItron: Foundational GUI Agent with Advanced Perception and Planning
• Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning
• Atom-Searcher: Enhancing Agentic Deep Research via Fine-Grained Atomic Thought Reward
• Careful Queries, Credible Results: Teaching RAG Models Advanced Web Search Tools with Reinforcement Learning
• One Token to Fool LLM-as-a-Judge
• MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environment
• WebSynthesis: World-Model-Guided MCTS for Efficient WebUI-Trajectory Synthesis
• From Web Search towards Agentic Deep Research: Incentivizing Search with Reasoning Agents
• TaPR: Test-Aware Policy Refinement for Feedback-Conditioned Code Generation
• The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning
• WebDancer: Towards Autonomous Information Seeking Agency
• WebCoT: Enhancing Web Agent Reasoning by Reconstructing Chain-of-Thought in Reflection, Branching, and Rollback
• SearchMaster: Grounded and Regulated Self-Play for Search Agents
• RMSWeb: Reflection, Failure-Mode Mining, and Salvage-DS for Web Agent Reinforcement Learning
• How Far Are LLMs from Professional Poker Players? Revisiting Game-Theoretic Reasoning with Agentic Tool Use
• From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms
• Autonomous Continual Learning for Environment Adaptation of Computer-Use Agents
• Escaping the Context Bottleneck: Active Context Curation for LLM Agents via Reinforcement Learning
• Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RL
• RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning
• CoCo: Code as CoT for Text-to-Image Preview and Rare Concept Generation

Related

• WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning
• WorkForceAgent-R1: Incentivizing Reasoning Capability in LLM-based Web Agents via Reinforcement Learning
• WebGym: Scaling Training Environments for Visual Web Agents with Realistic Tasks
• Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction
• Learning to Navigate the Web