STAGE: Simplified Text-Attributed Graph Embeddings Using Pre-trained LLMs

2024/07/10 by Aaron Zolnai-Lucas, Jack Boylan, Zolnai-Lucas, Aaron +5
Biochemistry, Genetics and Molecular Biology · Computer Science · #Advanced Graph Neural Networks #Artificial Intelligence (cs.AI) #Biomedical Text Mining and Ontologies #Computation and Language (cs.CL) #FOS: Computer and information sciences #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2407.12860

openalex publication_date 2024/07/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present Simplified Text-Attributed Graph Embeddings (STAGE), a straightforward yet effective method for enhancing node features in Graph Neural Network (GNN) models that encode Text-Attributed Graphs (TAGs). Our approach leverages Large-Language Models (LLMs) to generate embeddings for textual attributes. STAGE achieves competitive results on various node classification benchmarks while also maintaining a simplicity in implementation relative to current state-of-the-art (SoTA) techniques. We show that utilizing pre-trained LLMs as embedding generators provides robust features for ensemble GNN training, enabling pipelines that are simpler than current SoTA approaches which require multiple expensive training and prompting stages. We also implement diffusion-pattern GNNs in an effort to make this pipeline scalable to graphs beyond academic benchmarks.

Related

• LLM as GNN: Graph Vocabulary Learning for Text-Attributed Graph Foundation Models
• Can GNN be Good Adapter for LLMs?
• Prompt-based Node Feature Extractor for Few-shot Learning on Text-Attributed Graphs
• Can LLMs Convert Graphs to Text-Attributed Graphs?
• Efficient End-to-end Language Model Fine-tuning on Graphs