CAILA: Concept-Aware Intra-Layer Adapters for Compositional Zero-Shot Learning

2023/05/26 by Zheng, Zhaoheng, Zhu, Haidong, Nevatia, Ram · 5 citations
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2305.16681

Abstract

In this paper, we study the problem of Compositional Zero-Shot Learning (CZSL), which is to recognize novel attribute-object combinations with pre-existing concepts. Recent researchers focus on applying large-scale Vision-Language Pre-trained (VLP) models like CLIP with strong generalization ability. However, these methods treat the pre-trained model as a black box and focus on pre- and post-CLIP operations, which do not inherently mine the semantic concept between the layers inside CLIP. We propose to dive deep into the architecture and insert adapters, a parameter-efficient technique proven to be effective among large language models, into each CLIP encoder layer. We further equip adapters with concept awareness so that concept-specific features of "object", "attribute", and "composition" can be extracted. We assess our method on four popular CZSL datasets, MIT-States, C-GQA, UT-Zappos, and VAW-CZSL, which shows state-of-the-art performance compared to existing methods on all of them.

Cited by

• H²em: Learning Hierarchical Hyperbolic Embeddings for Compositional Zero-Shot Learning
• Language-conditioned world model improves policy generalization by reading environmental descriptions
• CAMS: Towards Compositional Zero-Shot Learning via Gated Cross-Attention and Multi-Space Disentanglement
• Talk in Pieces, See in Whole: Disentangling and Hierarchical Aggregating Representations for Language-based Object Detection
• A Conditional Probability Framework for Compositional Zero-shot Learning

Related

• Compositional Zero-Shot Learning with Contextualized Cues and Adaptive Contrastive Training
• Prompting Large Pre-trained Vision-Language Models For Compositional Concept Learning
• CAMS: Towards Compositional Zero-Shot Learning via Gated Cross-Attention and Multi-Space Disentanglement
• MSCI: Addressing CLIP's Inherent Limitations for Compositional Zero-Shot Learning
• Zero-Shot Compositional Concept Learning