Le Cam meets LeCun: Deficiency and Generic Feature Learning

2014/02/20 by Brendan van Rooyen, van Rooyen, Brendan, Robert C. Williamson +1
Computer Science · Mathematics · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (stat.ML) #Machine Learning and Algorithms #Machine Learning and Data Classification #stat.ML

paper · pdf · doi:10.48550/arxiv.1402.4884

25 pages, 2 figures

openalex publication_date 2014/02/20 · arxiv created 2014/02/21 · arxiv updated 2014/02/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

"Deep Learning" methods attempt to learn generic features in an unsupervised fashion from a large unlabelled data set. These generic features should perform as well as the best hand crafted features for any learning problem that makes use of this data. We provide a definition of generic features, characterize when it is possible to learn them and provide methods closely related to the autoencoder and deep belief network of deep learning. In order to do so we use the notion of deficiency and illustrate its value in studying certain general learning problems.

Citations

• Equivalent Comparisons of Experiments
• Laplacian Eigenmaps for Dimensionality Reduction and Data Representation
• Sparse coding with an overcomplete basis set: A strategy employed by V1?
• An Information-Maximization Approach to Blind Separation and Blind Deconvolution

Related

• A Theory of Feature Learning
• Feature learning is decoupled from generalization in high capacity neural networks
• Generalizing in the Real World with Representation Learning
• Long-Tailed Learning Requires Feature Learning
• Machine Learning vs Deep Learning: The Generalization Problem