CODA: Constructivism Learning for Instance-Dependent Dropout Architecture Construction

2021/06/15 by Xiaoli Li, Li, Xiaoli
Computer Science · Decision Sciences · Mathematics · Psychology · #Architecture #Artificial intelligence #Coda #Computer science #Constructivism (international relations) #Dropout (neural networks) #FOS: Computer and information sciences #Geology #History #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine learning #Model-Driven Software Engineering Techniques #Political science #Psychology #Scientific Computing and Data Management #Seismology #Software Engineering Research #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2106.08444

arxiv created 2021/06/15 · openalex publication_date 2021/06/15 · arxiv updated 2021/06/17 · openalex created_date 2021/06/22 · openalex updated_date 2026/07/28

Abstract

Dropout is attracting intensive research interest in deep learning as an efficient approach to prevent overfitting. Recently incorporating structural information when deciding which units to drop out produced promising results comparing to methods that ignore the structural information. However, a major issue of the existing work is that it failed to differentiate among instances when constructing the dropout architecture. This can be a significant deficiency for many applications. To solve this issue, we propose Constructivism learning for instance-dependent Dropout Architecture (CODA), which is inspired from a philosophical theory, constructivism learning. Specially, based on the theory we have designed a better drop out technique, Uniform Process Mixture Models, using a Bayesian nonparametric method Uniform process. We have evaluated our proposed method on 5 real-world datasets and compared the performance with other state-of-the-art dropout techniques. The experimental results demonstrated the effectiveness of CODA.

Citations

• A Bayesian encourages dropout
• Variational Dropout and the Local Reparameterization Trick
• Improving neural networks by preventing co-adaptation of feature detectors
• The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients
• Variational Dropout Sparsifies Deep Neural Networks
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Related

• Pushing the bounds of dropout
• A Bayesian encourages dropout
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Dropout with Tabu Strategy for Regularizing Deep Neural Networks
• Dropout: Explicit Forms and Capacity Control