Image Generation from Scene Graphs

2018/04/04 by Justin Johnson, Johnson, Justin, Agrim Gupta +3 · 1 voice · 26 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #cs.CV #cs.LG

paper · pdf · doi:10.48550/arxiv.1804.01622

To appear at CVPR 2018

arxiv created 2018/04/04 · openalex publication_date 2018/04/04 · arxiv published 2018/04/04 · arxiv updated 2018/04/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

To truly understand the visual world our models should be able not only to recognize images but also generate them. To this end, there has been exciting recent progress on generating images from natural language descriptions. These methods give stunning results on limited domains such as descriptions of birds or flowers, but struggle to faithfully reproduce complex sentences with many objects and relationships. To overcome this limitation we propose a method for generating images from scene graphs, enabling explicitly reasoning about objects and their relationships. Our model uses graph convolution to process input graphs, computes a scene layout by predicting bounding boxes and segmentation masks for objects, and converts the layout to an image with a cascaded refinement network. The network is trained adversarially against a pair of discriminators to ensure realistic outputs. We validate our approach on Visual Genome and COCO-Stuff, where qualitative results, ablations, and user studies demonstrate our method's ability to generate complex images with multiple objects.

Cited by

• CoSPlan: Corrective Sequential Planning via Scene Graph Incremental Updates
• SceneProp: Combining Neural Network and Markov Random Field for Scene-Graph Grounding
• Abstract 3D Perception for Spatial Intelligence in Vision-Language Models
• SG-CoT: An Ambiguity-Aware Robotic Planning Framework using Scene Graph Representations
• Unbiased Object Detection Beyond Frequency with Visually Prompted Image Synthesis
• MSGFusion: Multimodal Scene Graph-Guided Infrared and Visible Image Fusion
• Learning to Generate 4D LiDAR Sequences
• SATURN: Autoregressive Image Generation Guided by Scene Graphs
• CountLoop: Training-Free High-Instance Image Generation via Iterative Agent Guidance
• LoRAtorio: An intrinsic approach to LoRA Skill Composition
• LaRender: Training-Free Occlusion Control in Image Generation via Latent Rendering
• Optimization of Private Semantic Communication Performance: An Uncooperative Covert Communication Method
• LiDARCrafter: Dynamic 4D World Modeling from LiDAR Sequences
• La La LiDAR: Large-Scale Layout Generation from LiDAR Data
• Aether Weaver: Multimodal Affective Narrative Co-Generation with Dynamic Scene Graphs
• FROSS: Faster-than-Real-Time Online 3D Semantic Scene Graph Generation from RGB-D Images
• GIIFT: Graph-guided Inductive Image-free Multimodal Machine Translation
• Inversion-DPO: Precise and Efficient Post-Training for Diffusion Models
• From Vision To Language through Graph of Events in Space and Time: An Explainable Self-supervised Approach
• Generating Directed Graphs with Dual Attention and Asymmetric Encoding
• Graffe: Graph Representation Learning via Diffusion Probabilistic Models
• From Data to Modeling: Fully Open-vocabulary Scene Graph Generation
• Imagine, Verify, Execute: Memory-guided Agentic Exploration with Vision-Language Models
• SDIT: Scalable and Diverse Cross-domain Image Translation
• Using Scene Graph Context to Improve Image Generation
• MOC-GAN: Mixing Objects and Captions to Generate Realistic Images

Discussions

• Image Generation from Scene Graphs [hn, 2 points, 0 comments]

Related

• Learning Canonical Representations for Scene Graph to Image Generation
• Scene Graph Generation by Iterative Message Passing
• Interactive Image Generation Using Scene Graphs
• Generating Triples with Adversarial Networks for Scene Graph Construction
• Unconditional Scene Graph Generation