A Probabilistic Theory of Deep Learning

2015/04/02 by Ankit B. Patel, Ankit Patel, Tan Nguyen +4 · 1 voice · 62 citations
Computer Science · Mathematics · #Advanced Image and Video Retrieval Techniques #Artificial intelligence #Computer science #Convolutional neural network #Deep belief network #Deep learning #Discriminative model #Generative Adversarial Networks and Image Synthesis #Generative grammar #Generative model #Inference #Machine learning #Neural Networks and Applications #Probabilistic logic #cs.CV #cs.LG #cs.NE #stat.ML

paper · pdf · doi:10.48550/arxiv.1504.00641

published in arXiv (Cornell University) (Cornell University) · 56 pages, 6 figures, 2 tables

arxiv created 2015/04/02 · openalex publication_date 2015/04/02 · arxiv updated 2015/04/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

A grand challenge in machine learning is the development of computational algorithms that match or outperform humans in perceptual inference tasks that are complicated by nuisance variation. For instance, visual object recognition involves the unknown object position, orientation, and scale in object recognition while speech recognition involves the unknown voice pronunciation, pitch, and speed. Recently, a new breed of deep learning algorithms have emerged for high-nuisance inference tasks that routinely yield pattern recognition systems with near- or super-human capabilities. But a fundamental question remains: Why do they work? Intuitions abound, but a coherent framework for understanding, analyzing, and synthesizing deep learning architectures has remained elusive. We answer this question by developing a new probabilistic framework for deep learning based on the Deep Rendering Model: a generative probabilistic model that explicitly captures latent nuisance variation. By relaxing the generative model to a discriminative one, we can recover two of the current leading deep learning systems, deep convolutional neural networks and random decision forests, providing insights into their successes and shortcomings, as well as a principled route to their improvement.

Citations

• Gradient-based learning applied to document recognition

Cited by

• Between Homomorphic Signal Processing and Deep Neural Networks: Constructing Deep Algorithms for Polyphonic Music Transcription
• SLiMFast: Guaranteed Results for Data Fusion and Source Reliability
• Truncated Variational Expectation Maximization
• Use and usability: concepts of representation in philosophy, neuroscience, cognitive science, and computer science
• Latent Regression Bayesian Network for Data Representation
• A phase transition in diffusion models reveals the hierarchical nature of data
• Training Neural Networks Without Gradients: A Scalable ADMM Approach
• Deeply-Sparse Signal rePresentations (DS²P)
• A Simple Hierarchical Pooling Data Structure for Loop Closure
• Face representation by deep learning: a linear encoding in a parameter space?
• Semi-Supervised Learning with the Deep Rendering Mixture Model
• A Bayesian Perspective of Convolutional Neural Networks through a Deconvolutional Generative Model
• Visual Representations: Defining Properties and Deep Approximations
• PCANet: An energy perspective
• Deep Learning and Hierarchal Generative Models
• When are Deep Networks really better than Decision Forests at small sample sizes, and how?
• Self-supervised local learning rules learn the hidden hierarchical structure of high-dimensional data
• A Probabilistic Framework for Deep Learning

Discussions

• A Probabilistic Theory of Deep Learning [hn, 74 points, 20 comments]

Related

• A Probabilistic Framework for Deep Learning
• A Selective Overview of Deep Learning
• A Probabilistic Representation of Deep Learning
• Deep Learning: A Tutorial
• Probabilistic Deep Learning with Probabilistic Neural Networks and Deep Probabilistic Models