Experience-Driven Exploration for Efficient API-Free AI Agents
2025/10/17 by Tang, Chenwei, Xing, Jingyu, Liu, Xinyu +4
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences
paper · doi:10.48550/arxiv.2510.15259
Abstract
Most existing software lacks accessible Application Programming Interfaces (APIs), requiring agents to operate solely through pixel-based Graphical User Interfaces (GUIs). In this API-free setting, large language model (LLM)-based agents face severe efficiency bottlenecks: limited to local visual experiences, they make myopic decisions and rely on inefficient trial-and-error, hindering both skill acquisition and long-term planning. To address these challenges, we propose KG-Agent, an experience-driven learning framework that structures an agent's raw pixel-level interactions into a persistent State-Action Knowledge Graph (SA-KG). KG-Agent overcomes inefficient exploration by linking functionally similar but visually distinct GUI states, forming a rich neighborhood of experience that enables the agent to generalize from a diverse set of historical strategies. To support long-horizon reasoning, we design a hybrid intrinsic reward mechanism based on the graph topology, combining a state value reward for exploiting known high-value pathways with a novelty reward that encourages targeted exploration. This approach decouples strategic planning from pure discovery, allowing the agent to effectively value setup actions with delayed gratification. We evaluate KG-Agent in two complex, open-ended GUI-based decision-making environments (Civilization V and Slay the Spire), demonstrating significant improvements in exploration efficiency and strategic depth over the state-of-the-art methods.
Citations
- A Comprehensive Review of Multi-Agent Reinforcement Learning in Video Games
- LLM-based Agentic Reasoning Frameworks: A Survey from Methods to Scenarios
- OS Agents: A Survey on MLLM-based Agents for General Computing Devices Use
- Rethinking Agent Design: From Top-Down Workflows to Bottom-Up Skill Evolution
- UFO2: The Desktop AgentOS
- A Survey on (M)LLM-Based GUI Agents
- API Agents vs. GUI Agents: Divergence and Convergence
- DSGBench: A Diverse Strategic Game Benchmark for Evaluating LLM-based Agents in Complex Decision-Making Environments
- Optimus-2: Multimodal Minecraft Agent with Goal-Observation-Action Conditioned Policy
- UI-TARS: Pioneering Automated GUI Interaction with Native Agents
- Large Language Model-Brained GUI Agents: A Survey
- Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents
- From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
- LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead
- LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision, and the Road Ahead
- Cradle: Empowering Foundation Agents Towards General Computer Control
- UFO: A UI-Focused Agent for Windows OS Interaction
- Large Language Models Play StarCraft II: Benchmarks and A Chain of Summarization Approach
- CogAgent: A Visual Language Model for GUI Agents
- Foundation Models in Robotics: Applications, Challenges, and the Future
- Levels of AGI for Operationalizing Progress on the Path to AGI
- The Rise and Potential of Large Language Model Based Agents: A Survey
- AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation
- Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Control
- Voyager: An Open-Ended Embodied Agent with Large Language Models
- What does CLIP know about a red circle? Visual prompt engineering for VLMs
- Generative Agents: Interactive Simulacra of Human Behavior
- Micrograph segmentations for DDEVD
- Segment Anything
- GPT-4 Technical Report
- Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents
- Distributing Accountability, Not Capability: Phase Separation and the LLM Workflow Quadrant in Autonomous AI Agent Architectures
- Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
- Learning to Prompt for Vision-Language Models
- Learning Transferable Visual Models From Natural Language Supervision
- MetaAgents: Large Language Model Based Agents for Decision-Making on Teaming
Related