2021/12/08 by Kaifeng Gao, Long Chen, Gao, Kaifeng +8 · 2 citations
Computer Science · #Advanced Graph Neural Networks #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human Pose and Action Recognition #Multimedia (cs.MM) #Multimodal Machine Learning Applications #cs.CV #cs.MM
paper · pdf · doi:10.48550/arxiv.2112.04222
Accepted by CVPR 2022. Code is available at https://github.com/Dawn-LX/VidSGG-BIG. We also won the 1st place of Video Relation Understanding (VRU) Grand Challenge in ACM Multimedia 2021, with a simplified version of our model.(The code for object tracklets generation is available at https://github.com/Dawn-LX/VidVRD-tracklets)
openalex publication_date 2021/12/08 · arxiv created 2022/03/14 · arxiv updated 2022/03/15 · openalex created_date 2022/05/05 · openalex updated_date 2026/07/28
Today's VidSGG models are all proposal-based methods, i.e., they first generate numerous paired subject-object snippets as proposals, and then conduct predicate classification for each proposal. In this paper, we argue that this prevalent proposal-based framework has three inherent drawbacks: 1) The ground-truth predicate labels for proposals are partially correct. 2) They break the high-order relations among different predicate instances of a same subject-object pair. 3) VidSGG performance is upper-bounded by the quality of the proposals. To this end, we propose a new classification-then-grounding framework for VidSGG, which can avoid all the three overlooked drawbacks. Meanwhile, under this framework, we reformulate the video scene graphs as temporal bipartite graphs, where the entities and predicates are two types of nodes with time slots, and the edges denote different semantic roles between these nodes. This formulation takes full advantage of our new framework. Accordingly, we further propose a novel BIpartite Graph based SGG model: BIG. It consists of a classification stage and a grounding stage, where the former aims to classify the categories of all the nodes and the edges, and the latter tries to localize the temporal location of each relation instance. Extensive ablations on two VidSGG datasets have attested to the effectiveness of our framework and BIG. Code is available at https://github.com/Dawn-LX/VidSGG-BIG.