2020/12/07 by Yaniv Shulman, Shulman, Yaniv
Computer Science · Mathematics · Physics and Astronomy · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Model Reduction and Neural Networks #Neural Networks and Applications #cs.LG #stat.ML
paper · pdf · doi:10.48550/arxiv.2012.03653
openalex publication_date 2020/12/07 · arxiv created 2021/03/06 · arxiv updated 2021/03/09 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28
Modern neural network architectures typically have many millions of parameters and can be pruned significantly without substantial loss in effectiveness which demonstrates they are over-parameterized. The contribution of this work is two-fold. The first is a method for approximating a multivariate Bernoulli random variable by means of a deterministic and differentiable transformation of any real-valued multivariate random variable. The second is a method for model selection by element-wise multiplication of parameters with approximate binary gates that may be computed deterministically or stochastically and take on exact zero values. Sparsity is encouraged by the inclusion of a surrogate regularization to the L0 loss. Since the method is differentiable it enables straightforward and efficient learning of model architectures by an empirical risk minimization procedure with stochastic gradient descent and theoretically enables conditional computation during training. The method also supports any arbitrary group sparsity over parameters or activations and therefore offers a framework for unstructured or flexible structured model pruning. To conclude experiments are performed to demonstrate the effectiveness of the proposed approach.