2021/11/19 by Junhyeong Cho, Cho, Junhyeong, Youngseok Yoon +6 · 7 citations
Computer Science · Engineering · #Architecture #Artificial Intelligence (cs.AI) #Artificial intelligence #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Domain Adaptation and Few-Shot Learning #Encoder #Engineering #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Natural language processing #Noun #Pattern recognition (psychology) #Salient #Topic Modeling #Transformer #Verb #cs.AI #cs.CV #cs.LG
paper · pdf · doi:10.48550/arxiv.2111.10135
published in arXiv (Cornell University) (Cornell University) · Accepted to BMVC 2021, Code: https://github.com/jhcho99/gsrtr
arxiv created 2021/11/19 · openalex publication_date 2021/11/19 · arxiv updated 2021/11/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Grounded Situation Recognition (GSR) is the task that not only classifies a salient action (verb), but also predicts entities (nouns) associated with semantic roles and their locations in the given image. Inspired by the remarkable success of Transformers in vision tasks, we propose a GSR model based on a Transformer encoder-decoder architecture. The attention mechanism of our model enables accurate verb classification by capturing high-level semantic feature of an image effectively, and allows the model to flexibly deal with the complicated and image-dependent relations between entities for improved noun classification and localization. Our model is the first Transformer architecture for GSR, and achieves the state of the art in every evaluation metric on the SWiG benchmark. Our code is available at https://github.com/jhcho99/gsrtr .