Learning from LDA using Deep Neural Networks

2015/08/05 by Dongxu Zhang, Zhang, Dongxu, Tianyi Luo +5
Computer Science · #Topic Modeling #Natural Language Processing Techniques #Speech Recognition and Synthesis

paper · pdf · doi:10.48550/arxiv.1508.01011

Abstract

Latent Dirichlet Allocation (LDA) is a three-level hierarchical Bayesian model for topic inference. In spite of its great success, inferring the latent topic distribution with LDA is time-consuming. Motivated by the transfer learning approach proposed by~\newcitehinton2015distilling, we present a novel method that uses LDA to supervise the training of a deep neural network (DNN), so that the DNN can approximate the costly LDA inference with less computation. Our experiments on a document classification task show that a simple DNN can learn the LDA behavior pretty well, while the inference is speeded up tens or hundreds of times.

Related

• End-to-end Learning of LDA by Mirror-Descent Back Propagation over a Deep Architecture
• A New Approach to Speeding Up Topic Modeling
• Topic Analysis with Side Information: A Neural-Augmented LDA Approach
• A high-reproducibility and high-accuracy method for automated topic\n classification
• Supervised Topic Models