The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models

2020/12/12 by Tianlong Chen, Chen, Tianlong, Jonathan Frankle +11 · 12 citations
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural and Evolutionary Computing (cs.NE) #cs.CV #cs.LG #cs.NE

paper · pdf · doi:10.48550/arxiv.2012.06908

CVPR 2021

openalex publication_date 2020/12/12 · arxiv created 2021/03/29 · arxiv updated 2021/03/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The computer vision world has been re-gaining enthusiasm in various pre-trained models, including both classical ImageNet supervised pre-training and recently emerged self-supervised pre-training such as simCLR and MoCo. Pre-trained weights often boost a wide range of downstream tasks including classification, detection, and segmentation. Latest studies suggest that pre-training benefits from gigantic model capacity. We are hereby curious and ask: after pre-training, does a pre-trained model indeed have to stay large for its downstream transferability? In this paper, we examine supervised and self-supervised pre-trained models through the lens of the lottery ticket hypothesis (LTH). LTH identifies highly sparse matching subnetworks that can be trained in isolation from (nearly) scratch yet still reach the full models' performance. We extend the scope of LTH and question whether matching subnetworks still exist in pre-trained computer vision models, that enjoy the same downstream transfer performance. Our extensive experiments convey an overall positive message: from all pre-trained weights obtained by ImageNet classification, simCLR, and MoCo, we are consistently able to locate such matching subnetworks at 59.04% to 96.48% sparsity that transfer universally to multiple downstream tasks, whose performance see no degradation compared to using full pre-trained weights. Further analyses reveal that subnetworks found from different pre-training tend to yield diverse mask structures and perturbation sensitivities. We conclude that the core LTH observations remain generally relevant in the pre-training paradigm of computer vision, but more delicate discussions are needed in some cases. Codes and pre-trained models will be made available at: https://github.com/VITA-Group/CV_LTH_Pre-training.

Citations

• ImageNet Large Scale Visual Recognition Challenge
• An overview of gradient descent optimization algorithms
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal\n Networks
• VisDA: The Visual Domain Adaptation Challenge
• Learning deep representations by mutual information estimation and maximization
• The State of Sparsity in Deep Neural Networks
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Contrastive Multiview Coding
• Linear Mode Connectivity and the Lottery Ticket Hypothesis
• The Early Phase of Neural Network Training
• A Simple Framework for Contrastive Learning of Visual Representations
• Improved Baselines with Momentum Contrastive Learning
• Momentum Contrast for Unsupervised Visual Representation Learning
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Pruning Overparameterized Multi-Task Networks for Degraded Web Image Restoration
• SliceFine: The Universal Winning-Slice Hypothesis for Pretrained Networks
• Expand Neurons, Not Parameters
• Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization
• Pre-Trained Image Processing Transformer
• PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition
• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
• GANs Can Play Lottery Tickets Too
• BLaST: High Performance Inference and Pretraining using BLock Sparse Transformers
• Spending Your Winning Lottery Better After Drawing It
• Algorithm to Compilation Co-design: An Integrated View of Neural Network Sparsity
• A Unified Lottery Ticket Hypothesis for Graph Neural Networks
• SmartDeal: Re-Modeling Deep Network Weights for Efficient Inference and Training

Related

• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Self-supervised Pretraining of Visual Features in the Wild
• Are Large-scale Datasets Necessary for Self-Supervised Pre-training?
• Self-Supervised Pretraining Improves Self-Supervised Pretraining
• Rethinking ImageNet Pre-training