Simulating Environments with Reasoning Models for Agent Training

2025/11/03 by Yuetai Li, Huseyin A Inan, Li, Yuetai +15 · 3 citations
Computer Science · #Multi-Agent Systems and Negotiation #Intelligent Tutoring Systems and Adaptive Learning #AI-based Problem Solving and Planning

paper · pdf · doi:10.48550/arxiv.2511.01824

Abstract

LLM agents excel in compact environments requiring deep reasoning but remain brittle when operating in broader, more complex contexts that demand robustness across diverse tools and schemas. Building bespoke environments for training is heavy, brittle, and limits progress. In this paper, we demonstrate that LLMs can simulate realistic environment feedback without access to actual testbed data or APIs. Inspired by this capability, we propose two frameworks: Simia-SFT, a pipeline that synthesizes SFT data by amplifying small seed sets into diverse trajectories in an environment-agnostic manner, and Simia-RL, a framework that enables RL training without real environment implementations through LLM-simulated feedback. Fine-tuning open models yields consistent improvements across multiple benchmarks, surpassing GPT-4o and approaching o4-mini on τ²-Bench. Together, Simia-SFT and Simia-RL enable scalable agent training without environment engineering, replacing heavy and brittle implementations with flexible LLM-based simulation.

Citations

• τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment
• RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning
• APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay
• Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents
• Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuning
• ToolBridge: An Open-Source Dataset to Equip LLMs with External Tool Capabilities
• OfficeBench: Benchmarking Language Agents across Multiple Applications for Office Automation
• APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets
• τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
• LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models
• AgentTuning: Enabling Generalized Agent Abilities for LLMs
• Counterfactually Auditable Lifecycle Certification for Autonomous Agents
• Mind2Web: Towards a Generalist Agent for the Web
• ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases
• Gorilla: Large Language Model Connected with Massive APIs
• API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs
• WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
• LaMDA: Language Models for Dialog Applications

Cited by

• From Word to World: Can Large Language Models be Implicit Text-based World Models?
• AutoEnv: Automated Environments for Measuring Cross-Environment Agent Learning
• SynthTools: A Framework for Scaling Synthetic Tools for Agent Development

Related

• EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agents
• GEM: A Gym for Agentic LLMs
• Synthesizing Post-Training Data for LLMs through Multi-Agent Simulation
• Training LLM-Based Agents with Synthetic Self-Reflected Trajectories and Partial Masking
• AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning