On the challenges of learning with inference networks on sparse,\n high-dimensional data

2017/10/17 by Rahul G. Krishnan, Krishnan, Rahul G., Dawen Liang +4 · 2 citations
Computer Science · #Bayesian Methods and Mixture Models #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1710.06085

openalex publication_date 2017/10/17 · openalex created_date 2022/10/02 · openalex updated_date 2026/07/28

Abstract

We study parameter estimation in Nonlinear Factor Analysis (NFA) where the\ngenerative model is parameterized by a deep neural network. Recent work has\nfocused on learning such models using inference (or recognition) networks; we\nidentify a crucial problem when modeling large, sparse, high-dimensional\ndatasets -- underfitting. We study the extent of underfitting, highlighting\nthat its severity increases with the sparsity of the data. We propose methods\nto tackle it via iterative optimization inspired by stochastic variational\ninference citephoffman2013stochastic and improvements in the sparse data\nrepresentation used for inference. The proposed techniques drastically improve\nthe ability of these powerful models to fit sparse data, achieving\nstate-of-the-art results on a benchmark text-count dataset and excellent\nresults on the task of top-N recommendation.\n

Citations

• Importance Weighted Autoencoders
• Stochastic Backpropagation and Approximate Inference in Deep Generative\n Models
• "General Intelligence," Objectively Determined and Measured
• Variational Lossy Autoencoder

Cited by

• Mapping of Lesion Images to Somatic Mutations
• Universal Graph Learning for Power System Reconfigurations: Transfer Across Topology Variations

Related

• Merging Two Cultures: Deep and Statistical Learning
• Sparse Deep Learning: A New Framework Immune to Local Traps and\n Miscalibration
• VIKING: Deep variational inference with stochastic projections
• Just How Flexible are Neural Networks in Practice?
• Consistent Sparse Deep Learning: Theory and Computation