2025/04/23 by Yang Qi, Yang, Qi, Weichen Bi +6
Computer Science · #Advanced Image and Video Retrieval Techniques #Annotation #Artificial Intelligence (cs.AI) #Channel (broadcasting) #Computer Vision and Pattern Recognition (cs.CV) #Downstream (manufacturing) #FOS: Computer and information sciences #Feature (linguistics) #Graphical model #Graphical user interface #Human-Computer Interaction (cs.HC) #Metadata #Metric (unit) #Multimodal Machine Learning Applications #User interface #Web Data Mining and Analysis
paper · pdf · doi:10.48550/arxiv.2504.16419
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2025/04/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
Graphical User Interface (GUI) datasets are crucial for various downstream tasks. However, GUI datasets often generate annotation information through automatic labeling, which commonly results in inaccurate GUI element BBox annotations, including missing, duplicate, or meaningless BBoxes. These issues can degrade the performance of models trained on these datasets, limiting their effectiveness in real-world applications. Additionally, existing GUI datasets only provide BBox annotations visually, which restricts the development of visually related GUI downstream tasks. To address these issues, we introduce PixelWeb, a large-scale GUI dataset containing over 100,000 annotated web pages. PixelWeb is constructed using a novel automatic annotation approach that integrates visual feature extraction and Document Object Model (DOM) structure analysis through two core modules: channel derivation and layer analysis. Channel derivation ensures accurate localization of GUI elements in cases of occlusion and overlapping elements by extracting BGRA four-channel bitmap annotations. Layer analysis uses the DOM to determine the visibility and stacking order of elements, providing precise BBox annotations. Additionally, PixelWeb includes comprehensive metadata such as element images, contours, and mask annotations. Manual verification by three independent annotators confirms the high quality and accuracy of PixelWeb annotations. Experimental results on GUI element detection tasks show that PixelWeb achieves performance on the mAP95 metric that is 3-7 times better than existing datasets. We believe that PixelWeb has great potential for performance improvement in downstream tasks such as GUI generation and automated user interaction.