2025/02/12 by Zhao, Henry Hengyuan, Yang, Kaiming, Yu, Wendi +2 · 4 citations
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Embedded Systems Design Techniques #FOS: Computer and information sciences #Multiagent Systems (cs.MA) #Real-Time Systems Scheduling #Real-time simulation and control systems
paper · pdf · doi:10.48550/arxiv.2502.08047
openalex publication_date 2025/02/12 · openalex created_date 2025/02/15 · openalex updated_date 2026/07/28
Recent progress in GUI agents has substantially improved visual grounding, yet robust planning remains challenging, particularly when the environment deviates from a canonical initial state. In real applications, users often invoke assistance mid-workflow, where software may be partially configured, steps may have been executed in different orders, or the interface may differ from its default setup. Such task-state variability is pervasive but insufficiently evaluated in existing GUI benchmarks. To address this gap, we introduce WorldGUI, a benchmark covering ten widely used desktop and web applications with tasks instantiated under diverse, systematically constructed initial states. These variations capture realistic human-computer interaction settings and enable diagnostic evaluation of an agent's ability to recover, adapt plans, and handle non-default contexts. We further present WorldGUI-Agent, a simple and model-agnostic framework that organizes planning and execution around three critique stages, improving reliability in dynamic environments. Experiments demonstrate that state-of-the-art GUI agents exhibit substantial performance degradation under non-default initial conditions, revealing limited robustness and fragile planning behaviors. Our benchmark and framework provide a foundation for developing more adaptable and reliable GUI agents. The code and data are available at https://github.com/showlab/WorldGUI.