From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models

2024/04/01 by Rongjie Li, Songyang Zhang, Li, Rongjie +7 · 20 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2404.00906

openalex publication_date 2024/04/01 · openalex created_date 2024/04/03 · openalex updated_date 2026/07/28

Abstract

Scene graph generation (SGG) aims to parse a visual scene into an intermediate graph representation for downstream reasoning tasks. Despite recent advancements, existing methods struggle to generate scene graphs with novel visual relation concepts. To address this challenge, we introduce a new open-vocabulary SGG framework based on sequence generation. Our framework leverages vision-language pre-trained models (VLM) by incorporating an image-to-graph generation paradigm. Specifically, we generate scene graph sequences via image-to-text generation with VLM and then construct scene graphs from these sequences. By doing so, we harness the strong capabilities of VLM for open-vocabulary SGG and seamlessly integrate explicit relational modeling for enhancing the VL tasks. Experimental results demonstrate that our design not only achieves superior performance with an open vocabulary but also enhances downstream vision-language task performance through explicit relation modeling knowledge.

Cited by

• Generative Human-Object Interaction Detection via Differentiable Cognitive Steering of Multi-modal LLMs
• SpatialThinker: Reinforcing Scene Graph-Grounded Spatial Reasoning via Dense Rewards
• Interaction-Centric Knowledge Infusion and Transfer for Open-Vocabulary Scene Graph Generation
• Attention over Scene Graphs: Indoor Scene Representations Toward CSAI Classification
• CORE-3D: Context-aware Open-vocabulary Retrieval by Embeddings in 3D
• Reasoning in Computer Vision: Taxonomy, Models, Tasks, and Methodologies
• Open-Vocabulary HOI Detection with Interaction-aware Prompt and Concept Calibration
• Agent Identity Evals: Measuring Agentic Identity
• Bilateral Collaboration with Large Vision-Language Models for Open Vocabulary Human-Object Interaction Detection
• SPADE: Spatial-Aware Denoising Network for Open-vocabulary Panoptic Scene Graph Generation with Long- and Local-range Context Reasoning
• Synthetic Visual Genome
• DRAMA-X: A Fine-grained Intent Prediction and Risk Reasoning Benchmark For Driving
• Open World Scene Graph Generation using Vision Language Models
• From Data to Modeling: Fully Open-vocabulary Scene Graph Generation
• Harnessing Caption Detailness for Data-Efficient Text-to-Image Generation
• Modeling Scientific Experiment Scenes: Dataset and Model
• Relation-R1: Progressively Cognitive Chain-of-Thought Guided Reinforcement Learning for Unified Relation Comprehension
• Compile Scene Graphs with Reinforcement Learning
• Robo-SGG: Exploiting Layout-Oriented Normalization and Restitution Can Improve Robust Scene Graph Generation
• Generalized Visual Relation Detection with Diffusion Models

Related

• Open World Scene Graph Generation using Vision Language Models
• An Interpretable Model for Scene Graph Generation
• From Data to Modeling: Fully Open-vocabulary Scene Graph Generation
• Towards Open-vocabulary Scene Graph Generation with Prompt-based Finetuning
• Expanding Scene Graph Boundaries: Fully Open-vocabulary Scene Graph Generation via Visual-Concept Alignment and Retention