Sparsity-depth Tradeoff in Infinitely Wide Deep Neural Networks

2023/05/17 by Chun, Chanwoo, Lee, Daniel D.
#Disordered Systems and Neural Networks (cond-mat.dis-nn) #FOS: Biological sciences #FOS: Computer and information sciences #FOS: Physical sciences #Machine Learning (cs.LG) #Neurons and Cognition (q-bio.NC)

paper · doi:10.48550/arxiv.2305.10550

Abstract

We investigate how sparse neural activity affects the generalization performance of a deep Bayesian neural network at the large width limit. To this end, we derive a neural network Gaussian Process (NNGP) kernel with rectified linear unit (ReLU) activation and a predetermined fraction of active neurons. Using the NNGP kernel, we observe that the sparser networks outperform the non-sparse networks at shallow depths on a variety of datasets. We validate this observation by extending the existing theory on the generalization error of kernel-ridge regression.

Related

• The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective
• Deep Maxout Network Gaussian Process
• Deep Neural Networks as Gaussian Processes
• Wide Neural Networks with Bottlenecks are Deep Gaussian Processes
• Neural Networks with Sparse Activation Induced by Large Bias: Tighter Analysis with Bias-Generalized NTK