Learning Sparse Neural Networks via ℓ₀ and Tℓ₁ by a Relaxed Variable Splitting Method with Application to Multi-scale Curve Classification

2019/02/20 by Fanghui Xue, Jack Xin, Xue, Fanghui +1
Computer Science · Engineering · #Advanced Image Processing Techniques #FOS: Computer and information sciences #FOS: Mathematics #Image and Signal Denoising Methods #Machine Learning (cs.LG) #Medical Imaging and Analysis #Optimization and Control (math.OC)

paper · pdf · doi:10.48550/arxiv.1902.07419

openalex publication_date 2019/02/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We study sparsification of convolutional neural networks (CNN) by a relaxed variable splitting method of ℓ₀ and transformed-ℓ₁ (Tℓ₁) penalties, with application to complex curves such as texts written in different fonts, and words written with trembling hands simulating those of Parkinson's disease patients. The CNN contains 3 convolutional layers, each followed by a maximum pooling, and finally a fully connected layer which contains the largest number of network weights. With ℓ₀ penalty, we achieved over 99 % test accuracy in distinguishing shaky vs. regular fonts or hand writings with above 86 % of the weights in the fully connected layer being zero. Comparable sparsity and test accuracy are also reached with a proper choice of Tℓ₁ penalty.

Related

• Convergence of a Relaxed Variable Splitting Method for Learning Sparse Neural Networks via ℓ₁, ℓ₀, and transformed-ℓ₁ Penalties
• ℓ₀ Regularized Structured Sparsity Convolutional Neural Networks
• The State of Sparsity in Deep Neural Networks
• Winning the Lottery with Continuous Sparsification
• Embedding Differentiable Sparsity into Deep Neural Network