Interpretable Visual Understanding with Cognitive Attention Network

2021/08/06 by Tang, Xuejiao, Zhang, Wenbin, Yu, Yi +4 · 5 citations
#Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2108.02924

Abstract

While image understanding on recognition-level has achieved remarkable advancements, reliable visual scene understanding requires comprehensive image understanding on recognition-level but also cognition-level, which calls for exploiting the multi-source information as well as learning different levels of understanding and extensive commonsense knowledge. In this paper, we propose a novel Cognitive Attention Network (CAN) for visual commonsense reasoning to achieve interpretable visual understanding. Specifically, we first introduce an image-text fusion module to fuse information from images and text collectively. Second, a novel inference module is designed to encode commonsense among image, query and response. Extensive experiments on large-scale Visual Commonsense Reasoning (VCR) benchmark dataset demonstrate the effectiveness of our approach. The implementation is publicly available at https://github.com/tanjatang/CAN

Cited by

• GEM: Generative Entropy-Guided Preference Modeling for Few-shot Alignment of LLMs
• Fairness-Aware Graph Representation Learning with Limited Demographic Information
• Generative AI in Depth: A Survey of Recent Advances, Model Variants, and Real-World Applications
• Towards Transparent AI: A Survey on Explainable Language Models
• AMCR: A Framework for Assessing and Mitigating Copyright Risks in Generative Models

Related

• Attention Mechanism based Cognition-level Scene Understanding
• From Recognition to Cognition: Visual Commonsense Reasoning
• Cognitive Visual Commonsense Reasoning Using Dynamic Working Memory
• Learning to Agree on Vision Attention for Visual Commonsense Reasoning
• KVL-BERT: Knowledge Enhanced Visual-and-Linguistic BERT for Visual Commonsense Reasoning