Regularization for Unsupervised Deep Neural Nets

2016/08/15 by Baiyang Wang, Wang, Baiyang, Diego Klabjan +1
Computer Science · #Advanced Neural Network Applications #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Neural and Evolutionary Computing (cs.NE) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.1608.04426

openalex publication_date 2016/08/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Unsupervised neural networks, such as restricted Boltzmann machines (RBMs) and deep belief networks (DBNs), are powerful tools for feature selection and pattern recognition tasks. We demonstrate that overfitting occurs in such models just as in deep feedforward neural networks, and discuss possible regularization methods to reduce overfitting. We also propose a "partial" approach to improve the efficiency of Dropout/DropConnect in this scenario, and discuss the theoretical justification of these methods from model convergence and likelihood bounds. Finally, we compare the performance of these methods based on their likelihood and classification error rates for various pattern recognition data sets.

Related

• MaxDropout: Deep Neural Network Regularization Based on Maximum Output Values
• Ising-Dropout: A Regularization Method for Training and Compression of Deep Neural Networks
• Neural Network Regularization via Robust Weight Factorization
• Regularizing Deep Neural Networks by Noise: Its Interpretation and Optimization
• SoftTarget Regularization: An Effective Technique to Reduce Over-Fitting in Neural Networks