Efficient Deep Learning of GMMs

2019/02/15 by Shirin Jalali, Jalali, Shirin, Carl Nuzman +3 · 1 citation
Computer Science · #Bayesian Methods and Mixture Models #Speech Recognition and Synthesis #Music and Audio Processing

paper · pdf · doi:10.48550/arxiv.1902.05707

Abstract

We show that a collection of Gaussian mixture models (GMMs) in Rⁿ can be optimally classified using O(n) neurons in a neural network with two hidden layers (deep neural network), whereas in contrast, a neural network with a single hidden layer (shallow neural network) would require at least O(exp(n)) neurons or possibly exponentially large coefficients. Given the universality of the Gaussian distribution in the feature spaces of data, e.g., in speech, image and text, our result sheds light on the observed efficiency of deep neural networks in practical classification problems.

Cited by

• Depth separation beyond radial functions

Related

• Classification of Data Generated by Gaussian Mixture Models Using Deep ReLU Networks
• Classifying Overlapping Gaussian Mixtures in High Dimensions: From Optimal Classifiers to Neural Nets
• Deep Gaussian Mixture Models
• uGMM-NN: Univariate Gaussian Mixture Model Neural Network
• Rethinking Nonlinearity: Trainable Gaussian Mixture Modules for Modern Neural Architectures