How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervision

2022/04/11 by Dongkwan Kim, Alice Oh, Kim, Dongkwan +1 · 3 citations
Computer Science · Materials Science · #Advanced Graph Neural Networks #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning in Materials Science #Social and Information Networks (cs.SI) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2204.04879

openalex created_date 2021/02/01 · openalex publication_date 2022/04/11 · openalex updated_date 2026/07/28

Abstract

Attention mechanism in graph neural networks is designed to assign larger weights to important neighbor nodes for better representation. However, what graph attention learns is not understood well, particularly when graphs are noisy. In this paper, we propose a self-supervised graph attention network (SuperGAT), an improved graph attention model for noisy graphs. Specifically, we exploit two attention forms compatible with a self-supervised task to predict edges, whose presence and absence contain the inherent information about the importance of the relationships between nodes. By encoding edges, SuperGAT learns more expressive attention in distinguishing mislinked neighbors. We find two graph characteristics influence the effectiveness of attention forms and self-supervision: homophily and average degree. Thus, our recipe provides guidance on which attention design to use when those two graph characteristics are known. Our experiment on 17 real-world datasets demonstrates that our recipe generalizes across 15 datasets of them, and our models designed by recipe show improved performance over baselines.

Citations

• Deep Convolutional Networks on Graph-Structured Data
• Variational Graph Auto-Encoders
• Pitfalls of Graph Neural Network Evaluation
• Graph Neural Networks: A Review of Methods and Applications
• Fast Graph Representation Learning with PyTorch Geometric
• GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs
• Detecting Functionality-Specific Vulnerabilities via Retrieving Individual Functionality-Equivalent APIs in Open-Source Repositories
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Geom-GCN: Geometric Graph Convolutional Networks
• Measuring and Improving the Use of Graph Information in Graph Neural Networks
• Strategies for Pre-training Graph Neural Networks
• Open Graph Benchmark: Datasets for Machine Learning on Graphs
• Wiki-CS: A Wikipedia-Based Benchmark for Graph Neural Networks
• A Comprehensive Survey on Graph Neural Networks

Cited by

• Advancing Cache-Based Few-Shot Classification via Patch-Driven Relational Gated Graph Attention
• GFlowNets for Learning Better Drug-Drug Interaction Representations
• New Benchmarks for Learning on Non-Homophilous Graphs
• Gradient scarcity with Bilevel Optimization for Graph Learning
• Label-Wise Graph Convolutional Network for Heterophilic Graphs

Related

• Not All Neighbors Are Worth Attending to: Graph Selective Attention Networks for Semi-supervised Learning
• Understanding Attention and Generalization in Graph Neural Networks
• Supervised Attention Using Homophily in Graph Neural Networks
• Neighbor Overlay-Induced Graph Attention Network
• A Bird's-Eye Tutorial of Graph Attention Architectures