pix2code: Generating Code from a Graphical User Interface Screenshot

2017/05/22 by Tony Beltramelli, Beltramelli, Tony · 4 voices · 28 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Multimodal Machine Learning Applications #Video Analysis and Summarization #cs.AI #cs.CL #cs.CV #cs.LG #cs.NE

paper · pdf · doi:10.48550/arxiv.1705.07962

openalex publication_date 2017/05/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Transforming a graphical user interface screenshot created by a designer into computer code is a typical task conducted by a developer in order to build customized software, websites, and mobile applications. In this paper, we show that deep learning methods can be leveraged to train a model end-to-end to automatically generate code from a single input image with over 77% of accuracy for three different platforms (i.e. iOS, Android and web-based technologies).

Citations

• Recurrent Neural Network Regularization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Latent Predictor Networks for Code Generation

Cited by

• Design Theater: A Benchmark for Generative UI
• VSA:Visual-Structural Alignment for UI-to-Code
• Widget2Code: From Visual Widgets to UI Code via Multimodal LLMs
• Modular Layout Synthesis (MLS): Front-end Code via Structure Normalization and Constrained Generation
• Computer-Use Agents as Judges for Generative User Interface
• CrochetBench: Can Vision-Language Models Move from Describing to Doing in Crochet Domain?
• UI2Code^N: A Visual Language Model for Test-Time Scalable Interactive UI-to-Code Generation
• WebVIA: A Web-based Vision-Language Agentic Framework for Interactive and Verifiable UI-to-Code Generation
• PSD2Code: Automated Front-End Code Generation from Design Files via Multimodal Large Language Models
• VCode: a Multimodal Coding Benchmark with SVG as Symbolic Visual Representation
• SciTextures: Collecting and Connecting Visual Patterns, Models, and Code Across Science and Art
• WebGen-V Bench: Structured Representation for Enhancing Visual Design in LLM-based Web Generation and Evaluation
• WebRenderBench: Enhancing Web Interface Generation through Layout-Style Consistency and Reinforcement Learning
• IWR-Bench: Can LVLMs reconstruct interactive webpage from a user interaction video?
• UI-UG: A Unified MLLM for UI Understanding and Generation
• Automatically Generating Web Applications from Requirements Via Multi-Agent Test-Driven Development
• WebGen-Agent: Enhancing Interactive Website Generation with Multi-Level Feedback and Step-Level Reinforcement Learning
• UISim: An Interactive Image-Based UI Simulator for Dynamic Mobile Environments
• Improving User Interface Generation Models from Designer Feedback
• EfficientUICoder: Efficient MLLM-based UI Code Generation via Input and Output Token Compression
• WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generation
• Reverse Browser: Vector-Image-to-Code Generator
• Athena: Intermediate Representations for Iterative Scaffolded App Generation with an LLM
• Generative Interfaces for Language Models
• VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models
• LaTCoder: Converting Webpage Design to Code with Layout-as-Thought
• ScreenCoder: Advancing Visual-to-Code Generation for Front-End Automation via Modular Multimodal Agents
• Learning Only with Images: Visual Reinforcement Learning with Reasoning, Rendering, and Visual Feedback

Discussions

• Pix2code: Generating Code from a Graphical User Interface Screenshot [hn, 4 points, 1 comments]
• Pix2code: Generating Code from a Graphical User Interface Screenshot [hn, 2 points, 0 comments]
• pix2code: Generating Code from a Graphical User Interface Screenshot [hn, 2 points, 0 comments]
• Generating GUI code from screenshots using deep learning [hn, 1 points, 0 comments]

Related

• Automatically Generating Codes from Graphical Screenshots Based on Deep Autocoder
• Automatic code generation from sketches of mobile applications in end-user development using Deep Learning
• Reverse Browser: Vector-Image-to-Code Generator
• Automatic Graphics Program Generation using Attention-Based Hierarchical Decoder
• ScreenCoder: Advancing Visual-to-Code Generation for Front-End Automation via Modular Multimodal Agents