Let Your Graph Do the Talking: Encoding Structured Data for LLMs

2024/02/08 by Bryan Perozzi, Bahare Fatemi, Perozzi, Bryan +11 · 18 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · Decision Sciences · #Artificial Intelligence (cs.AI) #Biomedical Text Mining and Ontologies #FOS: Computer and information sciences #I.2.6 #I.2.7 #I.5.1 #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Scientific Computing and Data Management #Semantic Web and Ontologies #Social and Information Networks (cs.SI)

paper · pdf · doi:10.48550/arxiv.2402.05862

openalex publication_date 2024/02/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

How can we best encode structured data into sequential form for use in large language models (LLMs)? In this work, we introduce a parameter-efficient method to explicitly represent structured data for LLMs. Our method, GraphToken, learns an encoding function to extend prompts with explicit structured information. Unlike other work which focuses on limited domains (e.g. knowledge graph representation), our work is the first effort focused on the general encoding of structured data to be used for various reasoning tasks. We show that explicitly representing the graph structure allows significant improvements to graph reasoning tasks. Specifically, we see across the board improvements - up to 73% points - on node, edge and, graph-level tasks from the GraphQA benchmark.

Cited by

• Semantic Refinement with LLMs for Graph Representations
• Towards Efficient Hypergraph and Multi-LLM Agent Recommender Systems
• Efficiently Learning Branching Networks for Multitask Algorithmic Reasoning
• When Structure Doesn't Help: LLMs Do Not Read Text-Attributed Graphs as Effectively as We Expected
• Bridging the Divide: End-to-End Sequence-Graph Learning
• GraphShaper: Geometry-aware Alignment for Improving Transfer Learning in Text-Attributed Graphs
• Struc-EMB: The Potential of Structure-Aware Encoding in Language Embeddings
• Relational Transformer: Toward Zero-Shot Foundation Models for Relational Data
• Rethinking and Benchmarking Large Language Models for Graph Reasoning
• Benefits and Pitfalls of Reinforcement Learning for Language Model Planning: A Theoretical Perspective
• MIXRAG : Mixture-of-Experts Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering
• Actions Speak Louder than Prompts: A Large-Scale Study of LLMs for Graph Inference
• Self-Exploring Language Models for Explainable Link Forecasting on Temporal Graphs via Reinforcement Learning
• Efficient Graph Understanding with LLMs via Structured Context Injection
• A Graph Talks, But Who's Listening? Rethinking Evaluations for Graph-Language Models
• Multi-agent systems for chemical engineering: A review and perspective
• Harnessing Adaptive Topology Representations for Zero-Shot Graph Question Answering
• SAGE-HLS: Syntax-Aware AST-Guided LLM for High-Level Synthesis Code Generation

Related

• Talk like a Graph: Encoding Graphs for Large Language Models
• Can LLMs perform structured graph reasoning?
• Less is More: Learning Graph Tasks with Just LLMs
• When Structure Doesn't Help: LLMs Do Not Read Text-Attributed Graphs as Effectively as We Expected
• Efficient Graph Understanding with LLMs via Structured Context Injection