LayoutGPT: Compositional Visual Planning and Generation with Large Language Models

2023/05/24 by Weixi Feng, Feng, Weixi, Wanrong Zhu +15 · 96 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Artificial Intelligence (cs.AI) #Artificial intelligence #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Context (archaeology) #Controllability #Correctness #FOS: Computer and information sciences #Generative grammar #Generative model #Handwritten Text Recognition Techniques #Human–computer interaction #Image (mathematics) #Language model #Linguistics #Multimodal Machine Learning Applications #Programming language #Visual language

paper · pdf · doi:10.48550/arxiv.2305.15393

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2023/05/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Attaining a high degree of user controllability in visual generation often requires intricate, fine-grained inputs like layouts. However, such inputs impose a substantial burden on users when compared to simple text inputs. To address the issue, we study how Large Language Models (LLMs) can serve as visual planners by generating layouts from text conditions, and thus collaborate with visual generative models. We propose LayoutGPT, a method to compose in-context visual demonstrations in style sheet language to enhance the visual planning skills of LLMs. LayoutGPT can generate plausible layouts in multiple domains, ranging from 2D images to 3D indoor scenes. LayoutGPT also shows superior performance in converting challenging language concepts like numerical and spatial relations to layout arrangements for faithful text-to-image generation. When combined with a downstream image generation model, LayoutGPT outperforms text-to-image models/systems by 20-40% and achieves comparable performance as human users in designing visual layouts for numerical and spatial correctness. Lastly, LayoutGPT achieves comparable performance to supervised methods in 3D indoor scene synthesis, demonstrating its effectiveness and potential in multiple visual domains.

Cited by

• VULCAN: Tool-Augmented Multi Agents for Iterative 3D Object Arrangement
• I-Scene: 3D Instance Models are Implicit Generalizable Spatial Learners
• Exploring MLLM-Diffusion Information Transfer with MetaCanvas
• VDAWorld: World Modelling via VLM-Directed Abstraction and Simulation
• Learning by Analogy: A Causal Framework for Composition Generalization
• Phythesis: Physics-Guided Evolutionary Scene Synthesis for Energy-Efficient Data Center Design via LLMs
• Self-Evolving 3D Scene Generation from a Single Image
• BIM-Native Tokenization for Constraint-Aware Room Layout Synthesis
• PosterCopilot: Toward Layout Reasoning and Controllable Editing for Professional Graphic Design
• TabletopGen: Tabletop Scene Generation and Interactive Simulation for Robotic Manipulation
• GreenPlanner: Practical Floorplan Layout Generation via an Energy-Aware and Function-Feasible Generative Framework
• MarketGen: A Scalable Simulation Platform with Auto-Generated Embodied Supermarket Environments
• The Consistency Critic: Correcting Inconsistencies in Generated Images via Reference-Guided Attentive Alignment
• Yo'City: Personalized and Boundless 3D Realistic City Scene Generation via Self-Critic Expansion
• RoomPlanner: Explicit Layout Planner for Easier LLM-Driven 3D Room Generation
• DuoZone: A User-Centric, LLM-Guided Mixed-Initiative XR Window Management System
• A Two-Stage System for Layout-Controlled Image Generation using Large Language Models and Diffusion Models
• LEGO-Eval: Towards Fine-Grained Evaluation on Synthesizing 3D Embodied Environments with Tool Augmentation
• HiGS: Hierarchical Generative Scene Framework for Multi-Step Associative Semantic Spatial Composition
• StructLayoutFormer:Conditional Structured Layout Generation via Structure Serialization and Disentanglement
• PhyScensis: Physics-Augmented LLM Agents for Complex Physical Scene Arrangement
• Compositional Image Synthesis with Inference-Time Scaling
• GOPLA: Generalizable Object Placement Learning via Synthetic Augmentation of Human Arrangement
• FlashWorld: High-quality 3D Scene Generation within Seconds
• IL3D: A Large-Scale Indoor Layout Dataset for LLM-Driven 3D Scene Generation
• DisCo-Layout: Disentangling and Coordinating Semantic and Physical Refinement in a Multi-Agent Framework for 3D Indoor Layout Synthesis
• From Programs to Poses: Factored Real-World Scene Generation via Learned Program Libraries
• Automatic Text Box Placement for Supporting Typographic Design
• TTOM: Test-Time Optimization and Memorization for Compositional Video Generation
• MetaFind: Scene-Aware 3D Asset Retrieval for Coherent Metaverse Scene Generation
• MorphoSim: An Interactive, Controllable, and Editable Language-guided 4D World Simulator
• Stitch: Training-Free Position Control in Multimodal Diffusion Transformers
• IMG: Calibrating Diffusion Models via Implicit Multimodal Guidance
• M3DLayout: A Multi-Source Dataset of 3D Indoor Layouts and Structured Descriptions for 3D Generation
• MesaTask: Towards Task-Driven Tabletop Scene Generation via 3D Spatial Reasoning
• LayoutAgent: A Vision-Language Agent Guided Compositional Diffusion for Spatial Layout Planning
• SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent
• Blueprint-Bench: Comparing spatial intelligence of LLMs, agents and image models
• OverLayBench: A Benchmark for Layout-to-Image Generation with Dense Overlaps
• SPATIALGEN: Layout-guided 3D Indoor Scene Generation
• SceneGen: Single-Image 3D Scene Generation in One Feedforward Pass
• SemLayoutDiff: Semantic Layout Generation with Diffusion Model for Indoor Scene Synthesis
• SEAM: Semantically Equivalent Across Modalities Benchmark for Vision-Language Models
• Virtual Community: An Open World for Humans, Robots, and Society
• CountLoop: Training-Free High-Instance Image Generation via Iterative Agent Guidance
• ColorGPT: Leveraging Large Language Models for Multimodal Color Recommendation
• Canvas3D: Empowering Precise Spatial Control for Image Generation with Constraints from a 3D Virtual Canvas
• Bifrost-1: Bridging Multimodal LLMs and Diffusion Models with Patch-level CLIP Latents
• HOLODECK 2.0: Vision-Language-Guided 3D World Generation with Editing
• LiDARCrafter: Dynamic 4D World Modeling from LiDAR Sequences
• The Promise of RL for Autoregressive Image Editing
• HunyuanWorld 1.0: Generating Immersive, Explorable, and Interactive 3D Worlds from Words or Pixels
• FROSS: Faster-than-Real-Time Online 3D Semantic Scene Graph Generation from RGB-D Images
• Towards Geometric and Textural Consistency 3D Scene Generation via Single Image-guided Model Generation and Layout Optimization
• Test-time Prompt Refinement for Text-to-Image Models
• IGD: Instructional Graphic Design with Multimodal Layer Generation
• FloorplanQA: A Benchmark for Spatial Reasoning in LLMs using Structured Representations
• 3D-Generalist: Self-Improving Vision-Language-Action Models for Crafting 3D Worlds
• FurniMAS: Language-Guided Furniture Decoration using Multi-Agent System
• CoT-lized Diffusion: Let's Reinforce T2I Generation Step-by-step
• AutoLayout: Closed-Loop Layout Synthesis via Slow-Fast Collaborative Reasoning
• GeoProg3D: Compositional Visual Reasoning for City-Scale 3D Language Fields
• MusiXQA: Advancing Visual Music Understanding in Multimodal Large Language Models
• OracleFusion: Assisting the Decipherment of Oracle Bone Script with Structurally Constrained Semantic Typography
• Video Perception Models for 3D Scene Synthesis
• OptiScene: LLM-driven Indoor Scene Layout Generation via Scaled Human-aligned Data Synthesis and Multi-Stage Preference Optimization
• Programmable-Room: Interactive Textured 3D Room Meshes Generation Empowered by Large Language Models
• Visual-Instructed Degradation Diffusion for All-in-One Image Restoration
• Direct Numerical Layout Generation for 3D Indoor Scene Synthesis via Spatial Reasoning
• Lay-Your-Scene: Natural Scene Layout Generation with Diffusion Transformers
• PosterO: Structuring Layout Trees to Enable Language Models in Generalized Content-Aware Layout Generation
• ReSpace: Text-Driven 3D Indoor Scene Synthesis and Editing with Preference Alignment
• IMAGHarmony: Controllable Image Editing with Consistent Object Quantity and Layout
• PointT2I: LLM-based text-to-image generation via keypoints
• ArtiScene: Language-Driven Artistic 3D Scene Generation Through Image Intermediary
• ComposeAnything: Composite Object Priors for Text-to-Image Generation
• Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generation
• Fusion Intelligence for Digital Twinning AI Data Centers: A Synergistic GenAI-PhyAI Approach
• MCCD: Multi-Agent Collaboration-based Compositional Diffusion for Complex Text-to-Image Generation
• TextFlux: An OCR-Free DiT Model for High-Fidelity Multilingual Scene Text Synthesis
• GoT-R1: Unleashing Reasoning Capability of MLLM for Visual Generation with Reinforcement Learning
• Constructing a 3D Scene from a Single Image
• SG-Layout: Structured Scene Graph-Guided Layout Generation with LLMs
• MTVCrafter: 4D Motion Tokenization for Open-World Human Image Animation
• PlaceIt3D: Language-Guided Object Placement in Real 3D Scenes
• VSC: Visual Search Compositional Text-to-Image Diffusion Model
• Improving Editability in Image Generation with Layer-wise Memory
• Large Language Models are Universal Reasoners for Visual Generation
• CasaGPT: Cuboid Arrangement and Scene Assembly for Interior Design
• CoherenDream: Boosting Holistic Text Coherence in 3D Generation via Multimodal Large Language Models Feedback
• GeoMinLM: A Large Language Model in Geology and Mineral Survey in Yunnan Province
• ScanEdit: Hierarchically-Guided Functional 3D Scan Editing
• ESPLoRA: Enhanced Spatial Precision with Low-Rank Adaption in Text-to-Image Diffusion Models for High-Definition Synthesis
• iARCS: Iterative Agentic RL for Controllable 3D Scene Generation
• LayoutCoT: Unleashing the Deep Reasoning Potential of Large Language Models for Layout Generation
• Relation-Rich Visual Document Generator for Visual Information Extraction
• POEM: Precise Object-level Editing via MLLM control
• Storybooth: Training-free Multi-Subject Consistency for Improved Visual Storytelling

Related

• TextLap: Customizing Language Models for Text-to-Layout Planning
• VASCAR: Content-Aware Layout Generation via Visual-Aware Self-Correction
• LayoutPrompter: Awaken the Design Ability of Large Language Models
• Think, Plan, Paint: Layout-Aware Reasoning for Controllable Image Generation in Unified Models
• LayoutTransformer: Layout Generation and Completion with Self-attention