Colorization as a Proxy Task for Visual Understanding

2017/03/11 by Gustav Larsson, Larsson, Gustav, Michael Maire +3 · 56 citations
Computer Science · Engineering · #Advanced Chemical Sensor Technologies #Advanced Image and Video Retrieval Techniques #Artificial intelligence #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine learning #Proxy (statistics) #Segmentation #Task (project management) #cs.CV

paper · pdf · doi:10.48550/arxiv.1703.04044

published in arXiv (Cornell University) (Cornell University) · CVPR 2017 (Project page: http://people.cs.uchicago.edu/~larsson/color-proxy/)

openalex publication_date 2017/03/11 · arxiv created 2017/08/13 · arxiv updated 2017/08/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/08

Abstract

We investigate and improve self-supervision as a drop-in replacement for ImageNet pretraining, focusing on automatic colorization as the proxy task. Self-supervised training has been shown to be more promising for utilizing unlabeled data than other, traditional unsupervised learning methods. We build on this success and evaluate the ability of our self-supervised network in several contexts. On VOC segmentation and classification tasks, we present results that are state-of-the-art among methods not using ImageNet labels for pretraining representations. Moreover, we present the first in-depth analysis of self-supervision via colorization, concluding that formulation of the loss, training details and network architecture play important roles in its effectiveness. This investigation is further expanded by revisiting the ImageNet pretraining paradigm, asking questions such as: How much training data is needed? How many labels are needed? How much do features change when fine-tuned? We relate these questions back to self-supervision by showing that colorization provides a similarly powerful supervisory signal as various flavors of ImageNet pretraining.

Citations

• Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA
• Adversarial Feature Learning
• Learning Representations for Automatic Colorization
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Feedforward semantic segmentation with zoom-out features

Cited by

• Deep Graph Contrastive Representation Learning
• Label-efficient audio classification through multitask learning and self-supervision
• Conditional Alignment and Uniformity for Contrastive Learning with Continuous Proxy Labels
• Unsupervised Representation Learning by Predicting Image Rotations
• Learning to Look Around: Intelligently Exploring Unseen Environments for Unknown Tasks
• CO2: Consistent Contrast for Unsupervised Visual Representation Learning
• Unsupervised Feature Learning by Cross-Level Instance-Group Discrimination
• VELVET-Med: Vision and Efficient Language Pre-training for Volumetric Imaging Tasks in Medicine
• Exploit Clues from Views: Self-Supervised and Regularized Learning for Multiview Object Recognition
• Procedure Learning via Regularized Gromov-Wasserstein Optimal Transport
• Embedding Task Knowledge into 3D Neural Networks via Self-supervised Learning
• Fully Automatic Video Colorization with Self-Regularization and Diversity
• EASE: Embodied Active Event Perception via Self-Supervised Energy Minimization
• Temporal Interpolation as an Unsupervised Pretraining Task for Optical Flow Estimation
• Self-supervised Representation Learning with Local Aggregation for Image-based Profiling
• When Better Features Mean Greater Risks: The Performance-Privacy Trade-Off in Contrastive Learning
• Self-Supervised Feature Learning by Learning to Spot Artifacts
• Unsupervised Visual Attention and Invariance for Reinforcement Learning
• How PARTs assemble into wholes: Learning the relative composition of images
• Robust contrastive learning and nonlinear ICA in the presence of outliers
• gen2seg: Generative Models Enable Generalizable Instance Segmentation
• ROI Regularization for Semi-supervised and Supervised Learning
• ClusterFit: Improving Generalization of Visual Representations
• Unsupervised Domain Adaptation through Self-Supervision
• Improvements to Context Based Self-Supervised Learning

Related

• Discovery of Visual Semantics by Unsupervised and Self-Supervised Representation Learning
• How Useful is Self-Supervised Pretraining for Visual Tasks?
• When Does Contrastive Visual Representation Learning Work?
• Supervision Accelerates Pre-training in Contrastive Semi-Supervised Learning of Visual Representations
• Better Self-training for Image Classification through Self-supervision