Learning Compact Neural Networks with Regularization

2018/02/05 by Samet Oymak, Oymak, Samet · 1 citation
Computer Science · #Domain Adaptation and Few-Shot Learning #Machine Learning and ELM #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.1802.01223

Abstract

Proper regularization is critical for speeding up training, improving generalization performance, and learning compact models that are cost efficient. We propose and analyze regularized gradient descent algorithms for learning shallow neural networks. Our framework is general and covers weight-sharing (convolutional networks), sparsity (network pruning), and low-rank constraints among others. We first introduce covering dimension to quantify the complexity of the constraint set and provide insights on the generalization properties. Then, we show that proposed algorithms become well-behaved and local linear convergence occurs once the amount of data exceeds the covering dimension. Overall, our results demonstrate that near-optimal sample complexity is sufficient for efficient learning and illustrate how regularization can be beneficial to learn over-parameterized networks.

Cited by

• Communication-Efficient Edge AI: Algorithms and Systems

Related

• Learning Sparse Neural Networks through L₀ Regularization
• Regularization-based Pruning of Irrelevant Weights in Deep Neural Architectures
• Memory Bounded Deep Convolutional Networks
• Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guarantee
• Distance-Based Regularisation of Deep Networks for Fine-Tuning