Automatic Node Selection for Deep Neural Networks using Group Lasso Regularization

2016/11/17 by Tsubasa Ochiai, Shigeki Matsuda, Ochiai, Tsubasa +5 · 2 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Music and Audio Processing #Neural Networks and Applications #Speech Recognition and Synthesis

paper · pdf · doi:10.48550/arxiv.1611.05527

openalex publication_date 2016/11/17 · openalex created_date 2019/06/27 · openalex updated_date 2026/07/28

Abstract

We examine the effect of the Group Lasso (gLasso) regularizer in selecting the salient nodes of Deep Neural Network (DNN) hidden layers by applying a DNN-HMM hybrid speech recognizer to TED Talks speech data. We test two types of gLasso regularization, one for outgoing weight vectors and another for incoming weight vectors, as well as two sizes of DNNs: 2048 hidden layer nodes and 4096 nodes. Furthermore, we compare gLasso and L2 regularizers. Our experiment results demonstrate that our DNN training, in which the gLasso regularizer was embedded, successfully selected the hidden layer nodes that are necessary and sufficient for achieving high classification power.

Citations

• A Fast Learning Algorithm for Deep Belief Nets

Cited by

• Structured Output Regularization: a framework for few-shot transfer learning
• Structured Variational Learning of Bayesian Neural Networks with\n Horseshoe Priors

Related

• Graph based manifold regularized deep neural networks for automatic speech recognition
• Recurrent DNNs and its Ensembles on the TIMIT Phone Recognition Task
• Feature Learning in Deep Neural Networks - Studies on Speech Recognition\n Tasks
• Transferring Knowledge from a RNN to a DNN
• Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups