GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training

2020/09/07 by Tianle Cai, Cai, Tianle, Shengjie Luo +10 · 24 citations
Computer Science · Engineering · Materials Science · Mathematics · #Advanced Graph Neural Networks #FOS: Computer and information sciences #FOS: Mathematics #Ferroelectric and Negative Capacitance Devices #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning in Materials Science #Optimization and Control (math.OC) #cs.LG #math.OC #stat.ML

paper · pdf · doi:10.48550/arxiv.2009.03294

ICML 2021, Code: https://github.com/lsj2408/GraphNorm

openalex publication_date 2020/09/07 · openalex created_date 2020/09/11 · arxiv created 2021/06/11 · arxiv updated 2021/06/14 · openalex updated_date 2026/07/28

Abstract

Normalization is known to help the optimization of deep neural networks. Curiously, different architectures require specialized normalization methods. In this paper, we study what normalization is effective for Graph Neural Networks (GNNs). First, we adapt and evaluate the existing methods from other domains to GNNs. Faster convergence is achieved with InstanceNorm compared to BatchNorm and LayerNorm. We provide an explanation by showing that InstanceNorm serves as a preconditioner for GNNs, but such preconditioning effect is weaker with BatchNorm due to the heavy batch noise in graph datasets. Second, we show that the shift operation in InstanceNorm results in an expressiveness degradation of GNNs for highly regular graphs. We address this issue by proposing GraphNorm with a learnable shift. Empirically, GNNs with GraphNorm converge faster compared to GNNs using other normalization. GraphNorm also improves the generalization of GNNs, achieving better performance on graph classification benchmarks.

Citations

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Neural Message Passing for Quantum Chemistry
• MoleculeNet: A Benchmark for Molecular Machine Learning
• Detecting Functionality-Specific Vulnerabilities via Retrieving Individual Functionality-Equivalent APIs in Open-Source Repositories
• An Exponential Learning Rate Schedule for Deep Learning
• A Deep Learning Approach to Antibiotic Discovery
• Open Graph Benchmark: Datasets for Machine Learning on Graphs
• A Comprehensive Survey on Graph Neural Networks

Cited by

• ExPUFFIN: Thermodynamic Consistent Viscosity Prediction in an Extended Path-Unifying Feed-Forward Interfaced Network
• Learning the Cosmic Web: Graph-based Classification of Simulated Galaxies by their Dark Matter Environments
• Exploring Topological Bias in Heterogeneous Graph Neural Networks
• Language-Guided Graph Representation Learning for Video Summarization
• Free-Boundary Quasiconformal Maps via a Least-squares Operator in Diffeomorphism Optimization
• Rethinking Graph Super-resolution: Dual Frameworks for Topological Fidelity
• Surrogate Assisted Diversity Estimation in Neural Ensemble Search
• Morphology-Aware KOA Classification: Integrating Graph Priors with Vision Models
• Chaining 2-FWL GNNs for Combinatorial Graph Alignment
• Prompt-guided Disentangled Representation for Action Recognition
• Persistent Gaussian Perturbations Prevent Oversmoothing in Recurrent Graph Neural Networks
• On the Importance of Sampling in Training GCNs: Tighter Analysis and Variance Reduction
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• Wasserstein Embedding for Graph Learning
• Detecting AI Assistance in Abstract Complex Tasks
• MPP-GNN: Subject-Adaptive Community Detection for fMRI-Based Alzheimer's Disease Classification
• Latent Graph Representations for Critical View of Safety Assessment
• AGTCNet: A Graph-Temporal Approach for Principled Motor Imagery EEG Classification
• FuncGNN: Learning Functional Semantics of Logic Circuits with Graph Neural Networks
• Principled Data Augmentation for Learning to Solve Quadratic Programming Problems
• Information Obfuscation of Graph Neural Networks
• Nonparametric Teaching for Graph Property Learners
• Towards Unsupervised Training of Matching-based Graph Edit Distance Solver via Preference-aware GAN
• The Iterative Chainlet Partitioning Algorithm for the Traveling Salesman Problem with Drone and Neural Acceleration
• Multi-Level Attention Pooling for Graph Neural Networks: Unifying Graph Representations with Multiple Localities

Related

• Learning Graph Normalization for Graph Neural Networks
• FairNorm: Fair and Fast Graph Neural Network Training
• GRANOLA: Adaptive Normalization for Graph Neural Networks
• Bag of Tricks for Training Deeper Graph Neural Networks: A Comprehensive Benchmark Study
• ResNorm: Tackling Long-tailed Degree Distribution Issue in Graph Neural Networks via Normalization