Neural Autoregressive Distribution Estimation

2016/05/07 by Benigno Uria, Benigno Uría, Marc-Alexandre Côté +8 · 19 citations
Computer Science · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #cs.LG

paper · pdf · doi:10.48550/arxiv.1605.02226

openalex publication_date 2016/05/07 · arxiv created 2016/05/27 · arxiv updated 2016/05/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present Neural Autoregressive Distribution Estimation (NADE) models, which are neural network architectures applied to the problem of unsupervised distribution and density estimation. They leverage the probability product rule and a weight sharing scheme inspired from restricted Boltzmann machines, to yield an estimator that is both tractable and has good generalization performance. We discuss how they achieve competitive performance in modeling both binary and real-valued observations. We also present how deep NADE models can be trained to be agnostic to the ordering of input dimensions used by the autoregressive product rule decomposition. Finally, we also show how to exploit the topological structure of pixels in images using a deep convolutional architecture for NADE.

Citations

• Auto-Encoding Variational Bayes
• Connectionist learning of belief networks
• Generative Image Modeling Using Spatial LSTMs
• Training Products of Experts by Minimizing Contrastive Divergence
• A Fast Learning Algorithm for Deep Belief Nets
• ImageNet classification with deep convolutional neural networks
• Iterative Neural Autoregressive Distribution Estimator (NADE-k)
• Pixel Recurrent Neural Networks

Cited by

• Tracking large chemical reaction networks and rare events by neural networks
• Spanning Tree Autoregressive Visual Generation
• Likely Interpolants of Generative Models
• Black-Box Combinatorial Optimization with Order-Invariant Reinforcement Learning
• Efficient Autoregressive Inference for Transformer Probabilistic Models
• Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimization
• TABFAIRGDT: A Fast Fair Tabular Data Generator using Autoregressive Decision Trees
• Mining gold from implicit models to improve likelihood-free inference
• Ergodicity bounds in the Sliced Wasserstein distance for Schur stable autoregressive processes
• Privacy-Preserving Tabular Synthetic Data Generation Using TabularARGN
• Locally Masked Convolution for Autoregressive Models
• Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture
• Learning Distributions over Permutations and Rankings with Factorized Representations
• Automatic Music Transcription using Convolutional Neural Networks and Constant-Q transform
• Scalable Physics-Inspired Transformers for Spin Glasses
• Distilling Specialized Orders for Visual Generation
• A novel stellar spectrum denoising method based on deep Bayesian modeling
• Limitations of Autoregressive Models and Their Alternatives
• Quantum Error Mitigation with Diffusion-Like Models

Related

• A Deep and Tractable Density Estimator
• Iterative Neural Autoregressive Distribution Estimator (NADE-k)
• A Triangular Network For Density Estimation
• On the Equivalence Between Deep NADE and Generative Stochastic Networks
• Masked Autoregressive Flow for Density Estimation