Generalized Visual Relation Detection with Diffusion Models

2025/04/16 by Kaifeng Gao, Siqi Chen, Gao, Kaifeng +9
Computer Science · #Advanced Image and Video Retrieval Techniques #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Visual Attention and Saliency Detection

paper · pdf · doi:10.48550/arxiv.2504.12100

openalex publication_date 2025/04/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Visual relation detection (VRD) aims to identify relationships (or interactions) between object pairs in an image. Although recent VRD models have achieved impressive performance, they are all restricted to pre-defined relation categories, while failing to consider the semantic ambiguity characteristic of visual relations. Unlike objects, the appearance of visual relations is always subtle and can be described by multiple predicate words from different perspectives, e.g., ``ride'' can be depicted as ``race'' and ``sit on'', from the sports and spatial position views, respectively. To this end, we propose to model visual relations as continuous embeddings, and design diffusion models to achieve generalized VRD in a conditional generative manner, termed Diff-VRD. We model the diffusion process in a latent space and generate all possible relations in the image as an embedding sequence. During the generation, the visual and text embeddings of subject-object pairs serve as conditional signals and are injected via cross-attention. After the generation, we design a subsequent matching stage to assign the relation words to subject-object pairs by considering their semantic similarities. Benefiting from the diffusion-based generative process, our Diff-VRD is able to generate visual relations beyond the pre-defined category labels of datasets. To properly evaluate this generalized VRD task, we introduce two evaluation metrics, i.e., text-to-image retrieval and SPICE PR Curve inspired by image captioning. Extensive experiments in both human-object interaction (HOI) detection and scene graph generation (SGG) benchmarks attest to the superiority and effectiveness of Diff-VRD.

Citations

• Exploring Conditional Multi-Modal Prompts for Zero-shot HOI Detection
• From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models
• VDT: General-purpose Video Diffusion Transformers via Mask Modeling
• ViPLO: Vision Transformer based Pose-Conditioned Self-Loop Graph for Human-Object Interaction Detection
• CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matching
• Scalable Diffusion Models with Transformers
• Semantic-Conditional Diffusion Networks for Image Captioning
• Exploring Discrete Diffusion Models for Image Captioning
• DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models
• Understanding Diffusion Models: A Unified Perspective
• Towards Open-vocabulary Scene Graph Generation with Prompt-based Finetuning
• Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning
• Label Semantic Knowledge Distillation for Unbiased Scene Graph Generation
• NICEST: Noisy Label Correction and Training for Robust Scene Graph Generation
• Classifier-Free Diffusion Guidance
• Towards Hard-Positive Query Mining for DETR-based Human-Object Interaction Detection
• Guided-TTS 2: A Diffusion Model for High-quality Adaptive Text-to-Speech with Untranscribed Data
• Diffusion-LM Improves Controllable Text Generation
• End-to-End Zero-Shot HOI Detection via Vision and Language Knowledge Distillation
• Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model
• GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection
• Fine-Grained Scene Graph Generation with Data Transfer
• OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework
• RelTR: Relation Transformer for Scene Graph Generation
• SGTR: End-to-end Scene Graph Generation with Transformer
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• Efficient Two-Stage Detection of Human-Object Interactions with a Novel Unary-Pairwise Transformer
• Diffusion Autoencoders: Toward a Meaningful and Decodable Representation
• Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts
• Mining the Benefits of Two-stage and One-stage HOI Detection
• Variational Diffusion Models
• Diffusion Models Beat GANs on Image Synthesis
• Learning Transferable Visual Models From Natural Language Supervision
• Improved Denoising Diffusion Probabilistic Models
• Spatially Conditioned Graphs for Detecting Human-Object Interactions
• Open-Vocabulary Object Detection Using Captions
• Denoising Diffusion Implicit Models
• DiffWave: A Versatile Diffusion Model for Audio Synthesis
• CogTree: Cognition Tree Loss for Unbiased Scene Graph Generation
• PCPL: Predicate-Correlation Perception Learning for Unbiased Scene Graph Generation
• Debiased Contrastive Learning
• Denoising Diffusion Probabilistic Models
• Improved Techniques for Training Score-Based Generative Models
• End-to-End Object Detection with Transformers
• Unbiased Scene Graph Generation from Biased Training
• Detecting Human-Object Interactions via Functional Generalization
• Learning to Compose Dynamic Tree Structures for Visual Contexts
• Zero-shot Recognition via Semantic Embeddings and Knowledge Graphs
• Deep contextualized word representations
• Neural Motifs: Scene Graph Parsing with Global Context
• Attention Is All You Need
• Learning to Detect Human-Object Interactions
• Scene Graph Generation by Iterative Message Passing
• SPICE: Semantic Propositional Image Caption Evaluation
• Improving Variational Inference with Inverse Autoregressive Flow
• Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
• Ladder Variational Autoencoders
• Visual Semantic Role Labeling
• Glove: Global Vectors for Word Representation
• Microsoft COCO: Common Objects in Context
• Algorithms for the Assignment and Transportation Problems
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• DiffAugment: Diffusion based Long-Tailed Visual Relationship Recognition
• Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models
• Relation Rectification in Diffusion Model
• ReVersion: Diffusion-Based Relation Inversion from Images
• Knowledge-augmented Few-shot Visual Relation Detection