UserRL: Training Interactive User-Centric Agent via Reinforcement Learning

2025/09/24 by Cheng Qian, Zuxin Liu, Qian, Cheng +23 · 4 citations
Engineering · Psychology · #Action Observation and Synchronization #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Robot Manipulation and Learning #Social Robot Interaction and HRI

paper · pdf · doi:10.48550/arxiv.2509.19736

openalex publication_date 2025/09/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Reinforcement learning (RL) has shown promise in training agentic models that move beyond static benchmarks to engage in dynamic, multi-turn interactions. Yet, the ultimate value of such agents lies in their ability to assist users, a setting where diversity and dynamics of user interaction pose challenges. In this work, we propose UserRL, a unified framework for training and evaluating user-centric abilities through standardized gym environments paired with simulated users. We systematically vary turn-level reward assignment and trajectory-level score calculation to analyze how different formulations affect learning under the GRPO algorithm. Our experiments across Qwen3 models reveal three key findings: (i) SFT cold start is critical for unlocking initial interaction ability and enabling sustained RL improvements; (ii) deliberate trajectory scoring yields more efficient and effective multi-turn interactions; and (iii) while stronger simulated users (e.g., GPT-4o) facilitates training, open-source simulators (e.g., Qwen3-32B) remain a cost-effective and transferable option. Together, these results highlight that careful design of reward shaping and user simulation choice is as crucial as model scale, and establish UserRL as a practical pathway for developing robust user-centric agentic models. All codes and data are public for future research.

Citations

• MUA-RL: Multi-turn User-interacting Agent Reinforcement Learning for agentic tool use
• Training Long-Context, Multi-Turn Software Engineering Agents with Reinforcement Learning
• UserBench: An Interactive Gym Environment for User-Centric Agents
• From Web Search towards Agentic Deep Research: Incentivizing Search with Reasoning Agents
• τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment
• Position: Agent Should Invoke External Tools ONLY When Epistemically Necessary
• RM-R1: Reward Modeling as Reasoning
• Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learning
• RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning
• TTRL: Test-Time Reinforcement Learning
• Acting Less is Reasoning More! Teaching Model to Act Efficiently
• ToolRL: Reward is All Tool Learning Needs
• VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model
• VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
• MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents
• SMART: Self-Aware Agent for Tool Overuse Mitigation
• Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs
• SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance
• LAMBDA: A Large Model Based Data Agent
• τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
• Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Training
• SimPO: Simple Preference Optimization with a Reference-Free Reward
• Multi-turn Reinforcement Learning from Preference Human Feedback
• Aligning LLM Agents by Learning Latent Preference from User Edits
• A User-Centric Multi-Intent Benchmark for Evaluating Large Language Models
• ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models
• LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models
• On the Multi-turn Instruction Following for Conversational Web Agents
• Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agents
• WebLINX: Real-World Website Navigation with Multi-Turn Dialogue
• MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback
• The Rise and Potential of Large Language Model Based Agents: A Survey
• Counterfactually Auditable Lifecycle Certification for Autonomous Agents
• Mind2Web: Towards a Generalist Agent for the Web
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Toolformer: Language Models Can Teach Themselves to Use Tools
• Measuring and Narrowing the Compositionality Gap in Language Models
• Training language models to follow instructions with human feedback
• Proximal Policy Optimization Algorithms
• Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Reward Design
• Gymnasium: A Standard Interface for Reinforcement Learning Environments
• EscapeBench: Towards Advancing Creative Intelligence of Language Model Agents

Cited by

• CustomerSim: Benchmarking and Aligning Multimodal Language Models as Retail User Simulators
• Enterprise Deep Research: Steerable Multi-Agent Deep Research for Enterprise Analytics
• xRouter: Training Cost-Aware LLMs Orchestration System via Reinforcement Learning
• On Randomness in Agentic Evals

Related

• Improving Reinforcement Learning with Human Assistance: An Argument for Human Subject Studies with HIPPO Gym
• EasyRL: A Simple and Extensible Reinforcement Learning Framework
• Human AI interaction loop training: New approach for interactive reinforcement learning
• PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training
• GEM: A Gym for Agentic LLMs