Self-supervised Learning with Local Attention-Aware Feature

2021/08/01 by Trung X. Pham, Rusty John Lloyd Mina, Pham, Trung X. +5
Computer Science · #Advanced Image and Video Retrieval Techniques #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #FOS: Electrical engineering #Image and Video Processing (eess.IV) #Video Surveillance and Tracking Methods #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2108.00475

openalex publication_date 2021/08/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this work, we propose a novel methodology for self-supervised learning for generating global and local attention-aware visual features. Our approach is based on training a model to differentiate between specific image transformations of an input sample and the patched images. Utilizing this approach, the proposed method is able to outperform the previous best competitor by 1.03% on the Tiny-ImageNet dataset and by 2.32% on the STL-10 dataset. Furthermore, our approach outperforms the fully-supervised learning method on the STL-10 dataset. Experimental results and visualizations show the capability of successfully learning global and local attention-aware visual representations.

Citations

• The Pascal Visual Object Classes (VOC) Challenge
• Learning deep representations by mutual information estimation and maximization
• Learning Multiple Layers of Features from Tiny Images

Related

• Locally Enhanced Self-Attention: Combining Self-Attention and Convolution as Local and Context Terms
• Seeing the Whole in the Parts in Self-Supervised Representation Learning
• CASTing Your Model: Learning to Localize Improves Self-Supervised Representations
• Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks
• Explicitly Modeled Attention Maps for Image Classification