Topological Insights into Sparse Neural Networks

2020/06/24 by Shiwei Liu, Tim van der Lee, Liu, Shiwei +13
Computer Science · #Advanced Graph Neural Networks #Advanced Image and Video Retrieval Techniques #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Visual Attention and Saliency Detection

paper · doi:10.48550/arxiv.2006.14085

openalex publication_date 2020/06/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

Sparse neural networks are effective approaches to reduce the resource requirements for the deployment of deep neural networks. Recently, the concept of adaptive sparse connectivity, has emerged to allow training sparse neural networks from scratch by optimizing the sparse structure during training. However, comparing different sparse topologies and determining how sparse topologies evolve during training, especially for the situation in which the sparse structure optimization is involved, remain as challenging open questions. This comparison becomes increasingly complex as the number of possible topological comparisons increases exponentially with the size of networks. In this work, we introduce an approach to understand and compare sparse neural network topologies from the perspective of graph theory. We first propose Neural Network Sparse Topology Distance (NNSTD) to measure the distance between different sparse neural networks. Further, we demonstrate that sparse neural networks can outperform over-parameterized models in terms of performance, even without any further structure optimization. To the end, we also show that adaptive sparse connectivity can always unveil a plenitude of sparse sub-networks with very different topologies which outperform the dense model, by quantifying and comparing their topological evolutionary processes. The latter findings complement the Lottery Ticket Hypothesis by showing that there is a much more efficient and robust way to find "winning tickets". Altogether, our results start enabling a better theoretical understanding of sparse neural networks, and demonstrate the utility of using graph theory to analyze them.

Citations

• A distance measure between attributed relational graphs for pattern recognition
• ImageNet Large Scale Visual Recognition Challenge
• Variational Dropout Sparsifies Deep Neural Networks
• Exploring Sparsity in Recurrent Neural Networks
• Bayesian Compression for Deep Learning
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Grow and Prune Compact, Fast, and Accurate LSTMs
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Intrinsically Sparse Long Short-Term Memory Networks
• The State of Sparsity in Deep Neural Networks
• Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask
• Sparse Networks from Scratch: Faster Training without Losing Performance
• Deep Learning with S-shaped Rectified Linear Activation Units
• Learning Multiple Layers of Features from Tiny Images

Related

• Sparse Neural Networks Topologies
• NeuroFabric: Identifying Ideal Topologies for Training A Priori Sparse Networks
• Neural Network Topologies for Sparse Training
• Deconstructing the Structure of Sparse Neural Networks
• Understanding Sparse Neural Networks from their Topology via Multipartite Graph Representations