LLMs as Zero-shot Graph Learners: Alignment of GNN Representations with LLM Token Embeddings

2024/08/25 by Duo Wang, Wang, Duo, Yuan Zuo +5 · 24 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2408.14512

openalex publication_date 2024/08/25 · openalex created_date 2024/09/21 · openalex updated_date 2026/07/28

Abstract

Zero-shot graph machine learning, especially with graph neural networks (GNNs), has garnered significant interest due to the challenge of scarce labeled data. While methods like self-supervised learning and graph prompt learning have been extensively explored, they often rely on fine-tuning with task-specific labels, limiting their effectiveness in zero-shot scenarios. Inspired by the zero-shot capabilities of instruction-fine-tuned large language models (LLMs), we introduce a novel framework named Token Embedding-Aligned Graph Language Model (TEA-GLM) that leverages LLMs as cross-dataset and cross-task zero-shot learners for graph machine learning. Concretely, we pretrain a GNN, aligning its representations with token embeddings of an LLM. We then train a linear projector that transforms the GNN's representations into a fixed number of graph token embeddings without tuning the LLM. A unified instruction is designed for various graph tasks at different levels, such as node classification (node-level) and link prediction (edge-level). These design choices collectively enhance our method's effectiveness in zero-shot learning, setting it apart from existing methods. Experiments show that our graph token embeddings help the LLM predictor achieve state-of-the-art performance on unseen datasets and tasks compared to other methods using LLMs as predictors.

Cited by

• Temporal Visual Semantics-Induced Human Motion Understanding with Large Language Models
• When Structure Doesn't Help: LLMs Do Not Read Text-Attributed Graphs as Effectively as We Expected
• Empowering LLMs with Structural Role Inference for Zero-Shot Graph Learning
• GraphChain: Large Language Models for Large-scale Graph Analysis via Tool Chaining
• Large Language Models Meet Text-Attributed Graphs: A Survey of Integration Frameworks and Applications
• UniGTE: Unified Graph-Text Encoding for Zero-Shot Generalization across Graph Tasks and Domains
• H4G: Unlocking Faithful Inference for Zero-Shot Graph Learning in Hyperbolic Space
• Can Representation Gaps Be the Key to Enhancing Robustness in Graph-Text Alignment?
• GraphShaper: Geometry-aware Alignment for Improving Transfer Learning in Text-Attributed Graphs
• Preference-driven Knowledge Distillation for Few-shot Node Classification
• ReaLM: Residual Quantization Bridging Knowledge Graph Embeddings and Large Language Models
• FedBook: A Unified Federated Graph Foundation Codebook with Intra-domain and Inter-domain Knowledge Modeling
• Struc-EMB: The Potential of Structure-Aware Encoding in Language Embeddings
• GILT: An LLM-Free, Tuning-Free Graph Foundational Model for In-Context Learning
• Multimodal Large Language Models Meet Multimodal Emotion Recognition and Reasoning: A Survey
• One Prompt Fits All: Universal Graph Adaptation for Pretrained Models
• Graph-R1: Incentivizing the Zero-Shot Graph Learning Capability in LLMs via Explicit Reasoning
• AssemMate: Graph-Based LLM for Robotic Assembly Assistance
• A Graph Talks, But Who's Listening? Rethinking Evaluations for Graph-Language Models
• From Nodes to Narratives: Explaining Graph Neural Networks with LLMs and Graph Context
• Efficient Knowledge Probing of Large Language Models by Adapting Pre-trained Embeddings
• Entity Representation Learning Through Onsite-Offsite Graph for Pinterest Ads
• A Large Language Model Powered Integrated Circuit Footprint Geometry Understanding
• Ontology-Enhanced Knowledge Graph Completion using Large Language Models

Related

• Can GNN be Good Adapter for LLMs?
• Graph-R1: Incentivizing the Zero-Shot Graph Learning Capability in LLMs via Explicit Reasoning
• UniGTE: Unified Graph-Text Encoding for Zero-Shot Generalization across Graph Tasks and Domains
• GraphGPT: Graph Instruction Tuning for Large Language Models
• Exploring Graph Tasks with Pure LLMs: A Comprehensive Benchmark and Investigation