Graph Structure of Neural Networks

2020/07/13 by Jiaxuan You, Jure Leskovec, You, Jiaxuan +5 · 23 citations
Computer Science · Mathematics · #Advanced Graph Neural Networks #Algorithm #Artificial intelligence #Artificial neural network #Cluster analysis #Clustering coefficient #Computation #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Graph #Graph Theory and Algorithms #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Social and Information Networks (cs.SI) #Stochastic neural network #Theoretical computer science #Time delay neural network #cs.CV #cs.LG #cs.SI #stat.ML

paper · pdf · doi:10.48550/arxiv.2007.06559

published in arXiv (Cornell University) (Cornell University) · ICML 2020, with open-source code

openalex created_date 2020/06/19 · openalex publication_date 2020/07/13 · arxiv created 2020/08/27 · arxiv updated 2020/08/28 · openalex updated_date 2026/08/04

Abstract

Neural networks are often represented as graphs of connections between neurons. However, despite their wide use, there is currently little understanding of the relationship between the graph structure of the neural network and its predictive performance. Here we systematically investigate how does the graph structure of neural networks affect their predictive performance. To this end, we develop a novel graph-based representation of neural networks called relational graph, where layers of neural network computation correspond to rounds of message exchange along the graph structure. Using this representation we show that: (1) a "sweet spot" of relational graphs leads to neural networks with significantly improved predictive performance; (2) neural network's performance is approximately a smooth function of the clustering coefficient and average path length of its relational graph; (3) our findings are consistent across many different tasks and datasets; (4) the sweet spot can be identified efficiently; (5) top-performing neural networks have graph structure surprisingly similar to those of real biological neural networks. Our work opens new directions for the design of neural architectures and the understanding on neural networks in general.

Cited by

• Effective Fine-Tuning with Eigenvector Centrality Based Pruning
• Modeling Transformers as complex networks to analyze learning dynamics
• Application of discrete Ricci curvature in pruning randomly wired neural networks: A case study with chest x-ray classification of COVID-19
• Task complexity shapes internal representations and robustness in neural networks
• Graph Lineages and Skeletal Graph Products
• Self-similarity Analysis in Deep Neural Networks
• Effects of relational graph modularity and depth on the learning performance of neural networks
• NN-Former: Rethinking Graph Structure in Neural Architecture Representation
• On Interpretability of Artificial Neural Networks: A Survey
• Structure-Aware Automatic Channel Pruning by Searching with Graph Embedding
• CATE: Computation-aware Neural Architecture Encoding with Transformers
• Self-supervised Representation Learning for Evolutionary Neural Architecture Search
• Clusterability in Neural Networks
• Automatic and Structure-Aware Sparsification of Hybrid Neural ODEs
• A Second look at Exponential and Cosine Step Sizes: Simplicity, Adaptivity, and Performance
• Design Space for Graph Neural Networks
• sGrapp: Butterfly Approximation in Streaming Graphs
• Semantic Depth Matters: Explaining Errors of Deep Vision Networks through Perceived Class Similarities

Related

• Effects of relational graph modularity and depth on the learning performance of neural networks
• Theory of Graph Neural Networks: Representation and Learning
• Graph Neural Networks for Node-Level Predictions
• Relational Graph Neural Network Design via Progressive Neural Architecture Search
• Research on the application of graph data structure and graph neural network in node classification/clustering tasks