Location-Aware Self-Supervised Transformers for Semantic Segmentation

2022/12/05 by Mathilde Caron, Neil Houlsby, Caron, Mathilde +3 · 9 citations
Computer Science · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2212.02400

openalex publication_date 2022/12/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Pixel-level labels are particularly expensive to acquire. Hence, pretraining is a critical step to improve models on a task like semantic segmentation. However, prominent algorithms for pretraining neural networks use image-level objectives, e.g. image classification, image-text alignment a la CLIP, or self-supervised contrastive learning. These objectives do not model spatial information, which might be sub-optimal when finetuning on downstream tasks with spatial reasoning. In this work, we pretrain network with a location-aware (LOCA) self-supervised method which fosters the emergence of strong dense features. Specifically, we use both a patch-level clustering scheme to mine dense pseudo-labels and a relative location prediction task to encourage learning about object parts and their spatial arrangements. Our experiments show that LOCA pretraining leads to representations that transfer competitively to challenging and diverse semantic segmentation datasets.

Cited by

• Stable Single-Pixel Contrastive Learning for Semantic and Geometric Tasks
• FlowFeat: Pixel-Dense Embedding of Motion Profiles
• Enhancing Semantic Segmentation with Continual Self-Supervised Pre-training
• DIP: Unsupervised Dense In-Context Post-training of Visual Representations
• Position Prediction Self-Supervised Learning for Multimodal Satellite Imagery Semantic Segmentation
• How PARTs assemble into wholes: Learning the relative composition of images
• SPAR: Self-supervised Placement-Aware Representation Learning for Distributed Sensing
• Learning to Adapt to Position Bias in Vision Transformer Classifiers
• Self-Supervised Learning for Image Segmentation: A Comprehensive Survey

Related

• Generating Self-Guided Dense Annotations for Weakly Supervised Semantic Segmentation
• Improving Semantic Segmentation via Self-Training
• Self-supervised Pre-training for Semantic Segmentation in an Indoor Scene
• SIGN: Spatial-information Incorporated Generative Network for Generalized Zero-shot Semantic Segmentation
• Enhancing Semantic Segmentation with Continual Self-Supervised Pre-training