Building high-level features using large scale unsupervised learning

2011/12/29 by Quoc V. Le, Le, Quoc V., Marc’Aurelio Ranzato +14 · 1 voice · 20 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Face recognition and analysis #Machine Learning (cs.LG) #cs.LG

paper · pdf · doi:10.48550/arxiv.1112.6209

openalex publication_date 2011/12/29 · arxiv published 2011/12/29 · arxiv updated 2012/07/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We consider the problem of building high-level, class-specific feature detectors from only unlabeled data. For example, is it possible to learn a face detector using only unlabeled images? To answer this, we train a 9-layered locally connected sparse autoencoder with pooling and local contrast normalization on a large dataset of images (the model has 1 billion connections, the dataset has 10 million 200x200 pixel images downloaded from the Internet). We train this network using model parallelism and asynchronous SGD on a cluster with 1,000 machines (16,000 cores) for three days. Contrary to what appears to be a widely-held intuition, our experimental results reveal that it is possible to train a face detector without having to label images as containing a face or not. Control experiments show that this feature detector is robust not only to translation but also to scaling and out-of-plane rotation. We also find that the same network is sensitive to other high-level concepts such as cat faces and human bodies. Starting with these learned features, we trained our network to obtain 15.8% accuracy in recognizing 20,000 object categories from ImageNet, a leap of 70% relative improvement over the previous state-of-the-art.

Citations

• Emergence of Complex-Like Cells in a Temporal Product Network with Local Receptive Fields
• Gradient-based learning applied to document recognition
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Associative Memory using Attribute-Specific Neuron Groups-1: Learning between Multiple Cue Balls
• The brain-AI convergence: Predictive and generative world models for general-purpose computation
• Beyond Redundancy: Diverse and Specialized Multi-Expert Sparse Autoencoder
• Deep Roto-Translation Scattering for Object Classification
• Automatic Assessment of Students' Classroom Engagement with Bias Mitigated Multi-task Model
• Self-taught Object Localization with Deep Networks
• Fully Convolutional Networks for Semantic Segmentation
• Time-Aware Feature Selection: Adaptive Temporal Masking for Stable Sparse Autoencoder Training
• Associative Memory Model with Neural Networks: Memorizing multiple images with one neuron
• Deep Convolutional Features for Image Based Retrieval and Scene\n Categorization
• From Prediction to Understanding: Will AI Foundation Models Transform Brain Science?
• Unsupervised Feature Learning with C-SVDDNet
• Deep Extreme Feature Extraction: New MVA Method for Searching Particles in High Energy Physics
• Unsupervised Learning of Visual Representations using Videos
• Rank Ordered Autoencoders
• Variational Implicit Processes
• Learning to Predict Without Looking Ahead: World Models Without Forward Prediction
• PANDA: Pose Aligned Networks for Deep Attribute Modeling
• Learnable Pooling Regions for Image Classification
• Nonparametric Weight Initialization of Neural Networks via Integral Representation

Discussions

• Google research (with Jeff Dean): Face recognition with unsupervised learning [hn, 3 points, 0 comments]

Related

• Deep Trans-layer Unsupervised Networks for Representation Learning
• Learning Multiple Layers of Features from Tiny Images
• Unsupervised feature learning by augmenting single images
• Unsupervised Pre-Training of Image Features on Non-Curated Data
• Face Detection through Scale-Friendly Deep Convolutional Networks