A Theory of Feature Learning

2015/04/01 by Brendan van Rooyen, van Rooyen, Brendan, Robert C. Williamson +1
Computer Science · Mathematics · #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Generative Adversarial Networks and Image Synthesis #Image and Signal Denoising Methods #Machine Learning (cs.LG) #Machine Learning (stat.ML) #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1504.00083

arxiv created 2015/04/01 · openalex publication_date 2015/04/01 · arxiv updated 2015/04/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Feature Learning aims to extract relevant information contained in data sets in an automated fashion. It is driving force behind the current deep learning trend, a set of methods that have had widespread empirical success. What is lacking is a theoretical understanding of different feature learning schemes. This work provides a theoretical framework for feature learning and then characterizes when features can be learnt in an unsupervised fashion. We also provide means to judge the quality of features via rate-distortion theory and its generalizations.

Citations

• The information bottleneck method
• Sparse coding with an overcomplete basis set: A strategy employed by V1?

Related

• Feature learning is decoupled from generalization in high capacity neural networks
• A Theoretical Analysis on Feature Learning in Neural Networks: Emergence from Inputs and Advantage over Fixed Features
• Mitigating the Curse of Detail: Scaling Arguments for Feature Learning and Sample Complexity
• Le Cam meets LeCun: Deficiency and Generic Feature Learning
• Long-Tailed Learning Requires Feature Learning