GenEnv: Difficulty-Aligned Co-Evolution Between LLM Agents and Environment Simulators

2025/12/22 by Jiacheng Guo, Guo, Jiacheng, Ling Yang +15
Computer Science · Medicine · #Artificial Intelligence in Healthcare and Education #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Topic Modeling

paper · doi:10.48550/arxiv.2512.19682

openalex publication_date 2025/12/22 · openalex created_date 2025/12/24 · openalex updated_date 2026/07/28

Abstract

Training capable Large Language Model (LLM) agents is critically bottlenecked by the high cost and static nature of real-world interaction data. We address this by introducing GenEnv, a framework that establishes a difficulty-aligned co-evolutionary game between an agent and a scalable, generative environment simulator. Unlike traditional methods that evolve models on static datasets, GenEnv instantiates a dataevolving: the simulator acts as a dynamic curriculum policy, continuously generating tasks specifically tailored to the agent's ``zone of proximal development''. This process is guided by a simple but effective α-Curriculum Reward, which aligns task difficulty with the agent's current capabilities. We evaluate GenEnv on five benchmarks, including API-Bank, ALFWorld, BFCL, Bamboogle, and TravelPlanner. Across these tasks, GenEnv improves agent performance by up to +40.3% over 7B baselines and matches or exceeds the average performance of larger models. Compared to Gemini 2.5 Pro-based offline data augmentation, GenEnv achieves better performance while using 3.3× less data. By shifting from static supervision to adaptive simulation, GenEnv provides a data-efficient pathway for scaling agent capabilities.

Citations

• Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward
• Latent Collaboration in Multi-Agent Systems
• DeepAnalyze: Agentic Large Language Models for Autonomous Data Science
• LLMs as Scalable, General-Purpose Simulators For Evolving Digital Agent Training
• On the Role of Preference Variance in Preference Optimization
• Demystifying Reinforcement Learning in Agentic Reasoning
• Agent Learning via Early Experience
• Scaling Synthetic Task Generation for Agents via Exploration
• LIMI: Less is More for Agency
• MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agents
• Co-Evolving LLM Coder and Unit Tester via Reinforcement Learning
• LAM SIMULATOR: Advancing Data Generation for Large Action Model Training via Online Exploration and Trajectory Feedback
• Training LLM-Based Agents with Synthetic Self-Reflected Trajectories and Partial Masking
• Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPO
• Qwen3 Technical Report
• ComPO: Preference Alignment via Comparison Oracles
• A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Models
• Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
• STeCa: Step-level Trajectory Calibration for LLM Agent Learning
• Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agents
• AgentSociety: Large-Scale Simulation of LLM-Driven Generative Agents Advances Understanding of Human Behaviors and Society
• Agent-R: Training Language Model Agents to Reflect via Iterative Self-Training
• Qwen2.5 Technical Report
• AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials
• AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectories
• Good Data Is All Imitation Learning Needs
• The Llama 3 Herd of Models
• KnowAgent: Knowledge-Augmented Planning for LLM-Based Agents
• Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challenges
• LLM-DA: Data Augmentation via Large Language Models for Few-Shot Named Entity Recognition
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• TravelPlanner: A Benchmark for Real-World Planning with Language Agents
• Voyager: An Open-Ended Embodied Agent with Large Language Models
• API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs
• Reflexion: Language Agents with Verbal Reinforcement Learning
• Toolformer: Language Models Can Teach Themselves to Use Tools
• Understanding HTML with Large Language Models
• Measuring and Narrowing the Compositionality Gap in Language Models
• ReAct: Synergizing Reasoning and Acting in Language Models
• ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
• Trust Region Policy Optimization

Related

• AgentGym: Evolving Large Language Model-based Agents across Diverse Environments
• EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agents
• GEM: A Gym for Agentic LLMs
• AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation
• Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning