2020/03/10 by Daniel Filan, Shlomi Hod, Filan, Daniel +7 · 1 voice · 5 citations
Computer Science · Materials Science · #Artificial intelligence #Artificial neural network #Cluster analysis #Computer science #Deep neural networks #Machine Learning and ELM #Machine Learning in Materials Science #Machine learning #Modular design #Modular neural network #Modularity (biology) #Neural Networks and Applications #Pattern recognition (psychology) #Perceptron #Programming language #Pruning #Theoretical computer science #Time delay neural network #cs.LG #cs.NE
paper · pdf · doi:10.48550/arxiv.2003.04881
published in arXiv (Cornell University) (Cornell University) · 25 pages, 12 figures
openalex publication_date 2020/03/10 · arxiv created 2022/02/07 · arxiv updated 2022/02/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06
The learned weights of a neural network are often considered devoid of scrutable internal structure. To discern structure in these weights, we introduce a measurable notion of modularity for multi-layer perceptrons (MLPs), and investigate the modular structure of MLPs trained on datasets of small images. Our notion of modularity comes from the graph clustering literature: a "module" is a set of neurons with strong internal connectivity but weak external connectivity. We find that training and weight pruning produces MLPs that are more modular than randomly initialized ones, and often significantly more modular than random MLPs with the same (sparse) distribution of weights. Interestingly, they are much more modular when trained with dropout. We also present exploratory analyses of the importance of different modules for performance and how modules depend on each other. Understanding the modular structure of neural networks, when such structure exists, will hopefully render their inner workings more interpretable to engineers. Note that this paper has been superceded by "Clusterability in Neural Networks", arxiv:2103.03386 and "Quantifying Local Specialization in Deep Neural Networks", arxiv:2110.08058!