Self-Supervised Visual Representations Learning by Contrastive Mask Prediction

2021/08/18 by Yucheng Zhao, Zhao, Yucheng, Guangting Wang +7 · 4 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2108.07954

openalex publication_date 2021/08/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Advanced self-supervised visual representation learning methods rely on the instance discrimination (ID) pretext task. We point out that the ID task has an implicit semantic consistency (SC) assumption, which may not hold in unconstrained datasets. In this paper, we propose a novel contrastive mask prediction (CMP) task for visual representation learning and design a mask contrast (MaskCo) framework to implement the idea. MaskCo contrasts region-level features instead of view-level features, which makes it possible to identify the positive sample without any assumptions. To solve the domain gap between masked and unmasked features, we design a dedicated mask prediction head in MaskCo. This module is shown to be the key to the success of the CMP. We evaluated MaskCo on training datasets beyond ImageNet and compare its performance with MoCo V2. Results show that MaskCo achieves comparable performance with MoCo V2 using ImageNet training dataset, but demonstrates a stronger performance across a range of downstream tasks when COCO or Conceptual Captions are used for training. MaskCo provides a promising alternative to the ID-based methods for self-supervised learning in the wild.

Citations

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• YFCC100M
• Improved Baselines with Momentum Contrastive Learning
• Momentum Contrast for Unsupervised Visual Representation Learning
• Unsupervised Learning of Dense Visual Representations

Cited by

• Reducing Label Dependency in Human Activity Recognition with Wearables: From Supervised Learning to Novel Weakly Self-Supervised Approaches
• Conditional Representation Learning for Customized Tasks
• Beyond Instance Consistency: Investigating View Diversity in Self-supervised Learning
• Self-Supervised Contrastive Learning for Multi-Label Images

Related

• Improvements to Self-Supervised Representation Learning for Masked Image Modeling
• Masked Contrastive Representation Learning
• A Simple Framework for Contrastive Learning of Visual Representations
• PointCMP: Contrastive Mask Prediction for Self-supervised Learning on Point Cloud Videos
• ContrastMask: Contrastive Learning to Segment Every Thing