Computation on Sparse Neural Networks: an Inspiration for Future Hardware

2020/04/24 by Fei Sun, Minghai Qin, Sun, Fei +9 · 5 citations
Computer Science · Mathematics · #Advanced Neural Network Applications #Algorithm #Artificial intelligence #Artificial neural network #Computation #Computer engineering #Computer science #FOS: Computer and information sciences #Limit (mathematics) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Machine learning #Neural and Evolutionary Computing (cs.NE) #Perspective (graphical) #Software #Sparse matrix #Stochastic Gradient Optimization Techniques #cs.LG #cs.NE #stat.ML

paper · pdf · doi:10.48550/arxiv.2004.11946

published in ArXiv.org · 9 pages, 3 figures

arxiv created 2020/04/24 · openalex publication_date 2020/04/24 · arxiv updated 2020/04/28 · openalex created_date 2020/05/01 · openalex updated_date 2026/07/28

Abstract

Neural network models are widely used in solving many challenging problems, such as computer vision, personalized recommendation, and natural language processing. Those models are very computationally intensive and reach the hardware limit of the existing server and IoT devices. Thus, finding better model architectures with much less amount of computation while maximally preserving the accuracy is a popular research topic. Among various mechanisms that aim to reduce the computation complexity, identifying the zero values in the model weights and in the activations to avoid computing them is a promising direction. In this paper, we summarize the current status of the research on the computation of sparse neural networks, from the perspective of the sparse algorithms, the software frameworks, and the hardware accelerations. We observe that the search for the sparse structure can be a general methodology for high-quality model explorations, in addition to a strategy for high-efficiency model execution. We discuss the model accuracy influenced by the number of weight parameters and the structure of the model. The corresponding models are called to be located in the weight dominated and structure dominated regions, respectively. We show that for practically complicated problems, it is more beneficial to search large and sparse models in the weight dominated region. In order to achieve the goal, new approaches are required to search for proper sparse structures, and new sparse training hardware needs to be developed to facilitate fast iterations of sparse models.

Citations

• Playing Atari with Deep Reinforcement Learning
• Analysis of Individual Differences in Multidimensional Scaling Via an N-way Generalization of “Eckart-Young” Decomposition
• ImageNet classification with deep convolutional neural networks
• The Expression of a Tensor or a Polyadic as a Sum of Products
• A survey of sparse matrix-vector multiplication performance on large\n matrices
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Categorical Reparameterization with Gumbel-Softmax
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Delta Networks for Optimized Recurrent Network Computation
• Multimodal Machine Learning: A Survey and Taxonomy
• One Model To Learn Them All
• Dynamic Channel Pruning: Feature Boosting and Suppression
• Rethinking the Value of Network Pruning
• Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash
• CGaP: Continuous Growth and Pruning for Efficient Deep Learning
• Packing Sparse Convolutional Neural Networks for Efficient Systolic Array Implementations: Column Combining Under Joint Optimization
• PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Once-for-All: Train One Network and Specialize it for Efficient Deployment

Related

• Two Sparsities Are Better Than One: Unlocking the Performance Benefits of Sparse-Sparse Networks
• Truly Sparse Neural Networks at Scale
• The Sparsity Roofline: Understanding the Hardware Limits of Sparse Neural Networks
• A Theory of I/O-Efficient Sparse Neural Network Inference
• Training Sparse Neural Networks