Open World Scene Graph Generation using Vision Language Models

2025/06/09 by Amartya Dutta, Kazi Sajeed Mehrab, Dutta, Amartya +19 · 1 voice · 2 citations
Computer Science · #Embedding #Generative Adversarial Networks and Image Synthesis #Inference #Leverage (statistics) #Multimodal Machine Learning Applications #Pairwise comparison #Real world data #Salient #Scene graph #Topic Modeling #Variety (cybernetics) #cs.CL #cs.CV

paper · pdf · doi:10.48550/arxiv.2506.08189

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/06/09 · arxiv published 2025/06/09 · arxiv updated 2025/06/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Scene-Graph Generation (SGG) seeks to recognize objects in an image and distill their salient pairwise relationships. Most methods depend on dataset-specific supervision to learn the variety of interactions, restricting their usefulness in open-world settings, involving novel objects and/or relations. Even methods that leverage large Vision Language Models (VLMs) typically require benchmark-specific fine-tuning. We introduce Open-World SGG, a training-free, efficient, model-agnostic framework that taps directly into the pretrained knowledge of VLMs to produce scene graphs with zero additional learning. Casting SGG as a zero-shot structured-reasoning problem, our method combines multimodal prompting, embedding alignment, and a lightweight pair-refinement strategy, enabling inference over unseen object vocabularies and relation sets. To assess this setting, we formalize an Open-World evaluation protocol that measures performance when no SGG-specific data have been observed either in terms of objects and relations. Experiments on Visual Genome, Open Images V6, and the Panoptic Scene Graph (PSG) dataset demonstrate the capacity of pretrained VLMs to perform relational understanding without task-level training.

Citations

• PRISM-0: A Predicate-Rich Scene Graph Generation Framework for Zero-Shot Open-Vocabulary Tasks
• Relation-aware Hierarchical Prompt for Open-vocabulary Scene Graph Generation
• LLaVA-SpaceSGG: Visual Instruct Tuning for Open-vocabulary Scene Graph Generation with Enhanced Spatial Relations
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• Depth Anything V2
• Scene Graph Generation Strategy with Co-occurrence Knowledge and Learnable Term Frequency
• From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
• GPT4SGG: Synthesizing Scene Graphs from Holistic and Region-specific Narratives
• Enhancing Scene Graph Generation with Hierarchical Relationships and Commonsense Knowledge
• Expanding Scene Graph Boundaries: Fully Open-vocabulary Scene Graph Generation via Visual-Concept Alignment and Retention
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Visual Instruction Tuning
• Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open World
• Annotated Point Clouds and Images from a Spatial-Semantic Perception Pipeline: Robotized Deconstruction at the Reference Construction Site, Aachen
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• Panoptic Scene Graph Generation
• BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
• RelTR: Relation Transformer for Scene Graph Generation
• SGTR: End-to-end Scene Graph Generation with Transformer
• Structured Sparse R-CNN for Direct Scene Graph Generation
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• Learning Transferable Visual Models From Natural Language Supervision
• Open-Vocabulary Object Detection Using Captions
• Scene Graph Reasoning for Visual Question Answering
• GPS-Net: Graph Property Sensing Network for Scene Graph Generation
• Unbiased Scene Graph Generation from Biased Training
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Graphical Contrastive Losses for Scene Graph Parsing
• Learning to Compose Dynamic Tree Structures for Visual Contexts
• Neural Motifs: Scene Graph Parsing with Global Context
• Scene Graph Generation by Iterative Message Passing
• Graph-Structured Representations for Visual Question Answering
• Visual Relationship Detection with Language Priors
• Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
• Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
• Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond

Cited by

• VOST-SGG: VLM-Aided One-Stage Spatio-Temporal Scene Graph Generation
• Modeling Scientific Experiment Scenes: Dataset and Model

Discussions

• Open World Scene Graph Generation using Vision Language Models #DL #AI #ML #DeepLearning #ArtificialIntelligence #MachineLearning #ComputerVision #AutonomousVehicles #Robotics #LLM #VLM #LVLM arxiv.or [bsky, 1 points, 0 comments]

Related

• From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models
• Towards Open-vocabulary Scene Graph Generation with Prompt-based Finetuning
• Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open World
• An Interpretable Model for Scene Graph Generation
• Measuring Image-Relation Alignment: Reference-Free Evaluation of VLMs and Synthetic Pre-training for Open-Vocabulary Scene Graph Generation