Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction

2021/10/29 by Eli Chien, Chien, Eli, Wei-Cheng Chang +11 · 14 citations
Computer Science · Physics and Astronomy · #Advanced Graph Neural Networks #Complex Network Analysis Techniques #FOS: Computer and information sciences #Machine Learning (cs.LG) #Text and Document Classification Technologies #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2111.00064

openalex publication_date 2021/10/29 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28

Abstract

Learning on graphs has attracted significant attention in the learning community due to numerous real-world applications. In particular, graph neural networks (GNNs), which take numerical node features and graph structure as inputs, have been shown to achieve state-of-the-art performance on various graph-related learning tasks. Recent works exploring the correlation between numerical node features and graph structure via self-supervised learning have paved the way for further performance improvements of GNNs. However, methods used for extracting numerical node features from raw data are still graph-agnostic within standard GNN pipelines. This practice is sub-optimal as it prevents one from fully utilizing potential correlations between graph topology and node attributes. To mitigate this issue, we propose a new self-supervised learning framework, Graph Information Aided Node feature exTraction (GIANT). GIANT makes use of the eXtreme Multi-label Classification (XMC) formalism, which is crucial for fine-tuning the language model based on graph information, and scales to large datasets. We also provide a theoretical analysis that justifies the use of XMC over link prediction and motivates integrating XR-Transformers, a powerful method for solving XMC problems, into the GIANT framework. We demonstrate the superior performance of GIANT over the standard GNN pipeline on Open Graph Benchmark datasets: For example, we improve the accuracy of the top-ranked method GAMLP from 68.25% to 69.67%, SGC from 63.29% to 66.10% and MLP from 47.24% to 61.10% on the ogbn-papers100M dataset by leveraging GIANT.

Cited by

• Dual Refinement Cycle Learning: Unsupervised Text Classification of Mamba and Community Detection on Text Attributed Graph
• GraphMatch: Fusing Language and Graph Representations in a Dynamic Two-Sided Work Marketplace
• Odin: Oriented Dual-module Integration for Text-rich Network Representation Learning
• GCL-OT: Graph Contrastive Learning with Optimal Transport for Heterophilic Text-Attributed Graphs
• GRAPHTEXTACK: A Realistic Black-Box Node Injection Attack on LLM-Enhanced GNNs
• G2rammar: Bilingual Grammar Modeling for Enhanced Text-attributed Graph Learning
• Large Language Models Meet Text-Attributed Graphs: A Survey of Integration Frameworks and Applications
• Unveiling the Vulnerability of Graph-LLMs: An Interpretable Multi-Dimensional Adversarial Attack on TAGs
• GSTBench: A Benchmark Study on the Transferability of Graph Self-Supervised Learning
• A Graph Talks, But Who's Listening? Rethinking Evaluations for Graph-Language Models
• Graph is a Natural Regularization: Revisiting Vector Quantization for Graph Representation Learning
• HiTeC: Hierarchical Contrastive Learning on Text-Attributed Hypergraph with Semantic-Aware Augmentation
• Text-Attributed Graph Anomaly Detection via Multi-Scale Cross- and Uni-Modal Contrastive Learning
• Court of LLMs: Evidence-Augmented Generation via Multi-LLM Collaboration for Text-Attributed Graph Anomaly Detection

Related

• Bag of Tricks for Node Classification with Graph Neural Networks
• Large-scale graph representation learning with very deep GNNs and self-supervision
• ScaleGNN: Towards Scalable Graph Neural Networks via Adaptive High-order Neighboring Feature Fusion
• GraphFM: Improving Large-Scale GNN Training via Feature Momentum
• GaGSL: Global-augmented Graph Structure Learning via Graph Information Bottleneck