Transformers are Graph Neural Networks

2025/06/27 by Chaitanya K. Joshi, Joshi, Chaitanya K. · 6 voices · 21 citations
Computer Science · Engineering · #Advanced Graph Neural Networks #Ferroelectric and Negative Capacitance Devices #Graph Theory and Algorithms #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.2506.22084

openalex publication_date 2025/06/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We establish connections between the Transformer architecture, originally introduced for natural language processing, and Graph Neural Networks (GNNs) for representation learning on graphs. We show how Transformers can be viewed as message passing GNNs operating on fully connected graphs of tokens, where the self-attention mechanism capture the relative importance of all tokens w.r.t. each-other, and positional encodings provide hints about sequential ordering or structure. Thus, Transformers are expressive set processing networks that learn relationships among input elements without being constrained by apriori graphs. Despite this mathematical connection to GNNs, Transformers are implemented via dense matrix operations that are significantly more efficient on modern hardware than sparse message passing. This leads to the perspective that Transformers are GNNs currently winning the hardware lottery.

Citations

• On the Opportunities and Risks of Foundation Models
• Relational inductive biases, deep learning, and graph networks
• Attention Is All You Need
• Deep Residual Learning for Image Recognition
• Neural Machine Translation by Jointly Learning to Align and Translate

Cited by

• Transformers are Bayesian Networks
• Iterative Compositional Data Generation for Robot Control
• How Smoothing is N-simplicial Attention?
• Solving Oversmoothing in GNNs via Nonlocal Message Passing: Algebraic Smoothing and Depth Scalability
• Understanding the Failure Modes of Transformers through the Lens of Graph Neural Networks
• Multi-Scale Protein Structure Modelling with Geometric Graph U-Nets
• CNN on `Top': In Search of Scalable & Lightweight Image-based Jet Taggers
• Learning what to say and how precisely: Efficient Communication via Differentiable Discrete Communication Learning
• Enhancing Graph Representations with Neighborhood-Contextualized Message-Passing
• A Graph Signal Processing Framework for Hallucination Detection in Large Language Models
• Fighter: Unveiling the Graph Convolutional Nature of Transformers in Time Series Modeling
• VideoNSA: Native Sparse Attention Scales Video Understanding
• The Role of Feature Interactions in Graph-based Tabular Deep Learning
• FieldFormer: Locality-Aware Transformers for Spatio-Temporal Modeling on Sparse Sensor Networks
• Transformers Can Learn Connectivity in Some Graphs but Not Others
• Graph Variate Neural Networks
• An Attention-Based Stochastic Simulator for Multisite Extremes to Evaluate Nonstationary, Cascading Flood Risk
• A Design Co-Pilot for Task-Tailored Manipulators
• Dynamic Relational Priming Improves Transformer in Multivariate Time Series
• Latent Graph Learning in Generative Models of Neural Signals
• Molecular Machine Learning in Chemical Process Design

Discussions

• Transformers Are Graph Neural Networks [hn, 33 points, 2 comments]
• Needing a permanent doi was a forcing function to revisit this article on Transformers and Graph Neural Networks: arxiv.org/abs/2506.22084 This is (very) old news, so what's new? [bsky, 6 points, 1 comments]
• This is interesting, especially bc I understand GNN more than transformers arxiv.org/abs/2506.22084 [bsky, 2 points, 0 comments]
• Transformers are graph neural networks applied to fully-connected token graphs. Interesting perspective though it is natural as the graph attention networks adopt transformer architecture. [bsky, 2 points, 0 comments]
• I realize that I've been sensitized to all the chatter, but it could be more like a graph neural network with attention. Though maybe on the sparse side , rather than dense? Anyway, this is what got m [bsky, 0 points, 1 comments]
• I mean there is even a pednatic point here cause apparentily some argue transformers are GNNs arxiv.org/pdf/2506.22084 [bsky, 0 points, 0 comments]

Related

• Pure Transformers are Powerful Graph Learners
• Transformers as Graph-to-Graph Models
• A Generalization of Transformer Networks to Graphs
• Plain Transformers Can be Powerful Graph Learners
• GraphiT: Encoding Graph Structure in Transformers