Mixtures of Sparse Autoregressive Networks

2015/11/15 by Marc Goessling, Goessling, Marc, Yali Amit +1
Computer Science · #Bayesian Methods and Mixture Models #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · pdf · doi:10.48550/arxiv.1511.04776

openalex publication_date 2015/11/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We consider high-dimensional distribution estimation through autoregressive networks. By combining the concepts of sparsity, mixtures and parameter sharing we obtain a simple model which is fast to train and which achieves state-of-the-art or better results on several standard benchmark datasets. Specifically, we use an L1-penalty to regularize the conditional distributions and introduce a procedure for automatic parameter sharing between mixture components. Moreover, we propose a simple distributed representation which permits exact likelihood evaluations since the latent variables are interleaved with the observable variables and can be easily integrated out. Our model achieves excellent generalization performance and scales well to extremely high dimensions.

Citations

• A tutorial on hidden Markov models and selected applications in speech recognition
• Representation Learning: A Review and New Perspectives
• Sum-Product Networks: A New Deep Architecture

Related

• Support Recovery in Mixture Models with Sparse Parameters
• Learning Sparse Mixture Models
• Mix-nets: Factored Mixtures of Gaussians in Bayesian Networks With Mixed Continuous And Discrete Variables
• Graph-Regularized Learning of Gaussian Mixture Models
• High dimensional Sparse Gaussian Graphical Mixture Model