LLM Features Can Hurt GNNs: Concatenation Interference on Homophilous Graph Benchmarks

2026/06/16 by Zhongyuan Wang, Pratyusha Vemuri · 1 voice
Computer Science · #cs.LG #cs.AI #cs.CL #cs.SI

paper · pdf

Abstract

Adding LLM-generated node features to graph neural networks (GNNs) is widely reported to improve accuracy on standard benchmarks. We document a contrasting observation: when LLM features are introduced through pure input concatenation (rather than joint training, distillation, or prompt-conditioning), they can systematically degrade accuracy on the same homophilous benchmarks where end-to-end LLM pipelines succeed. With an MLP backbone on the Planetoid public split and bag-of-words original features, concatenating SBERT-encoded GPT-4o-mini TAPE features reduces PubMed test accuracy by -17.0 +/- 0.3 pp and Cora by -4.3 +/- 0.6 pp (CiteSeer -0.6 +/- 0.8 pp, within seed noise). The drop attenuates as we relax each condition (GCN / GCNII / GAT backbones, random splits, smaller encoders) and reverses on medium-homophily WikiCS (+4.4 pp) and ogbn-arxiv (+11.7 pp). To predict when concatenation helps versus hurts, we report a simple measure of LLM-alone discriminability, Delta_sig. Across 9 datasets Delta_sig correlates with the concatenation cost more strongly than homophily at point estimate (r² = 0.38 vs. 0.06; N=9, bootstrap CIs overlap). The bootstrap-best change-point is tau = 13.8 pp, and the rule "Delta_sig <= tau predicts non-positive concat cost" classifies 7/9 datasets correctly; since 60% of bootstrap samples place tau in [5, 30] pp, we treat Delta_sig as an interpretive lens rather than a precision filter. A dimension-controlled ablation on PubMed places the LLM-feature drop between same-source PCA (-2.3 pp) and same-dim Gaussian noise (-37.3 pp), ruling out dimensionality and weight-decay artifacts. Nine PubMed configurations fit a power law |Delta_concat| proportional to (sqrt(d_l/n))¹.31 with r² = 0.97; the low-Delta_sig, small-n corner is exactly where the headline -17 pp PubMed deficit appears.

Citations

• When Structure Doesn't Help: LLMs Do Not Read Text-Attributed Graphs as Effectively as We Expected
• A Closer Look at Multimodal Representation Collapse
• When Do LLMs Help With Node Classification? A Comprehensive Analysis
• Exact Computation of Any-Order Shapley Interactions for Graph Neural Networks
• GL-Fusion: Rethinking the Combination of Graph Neural Network and Large Language model
• GLBench: A Comprehensive Benchmark for Graph with Large Language Models
• Text-space Graph Foundation Models: Comprehensive Benchmarks and New Insights
• LLaGA: Large Language and Graph Assistant
• NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
• GraphGPT: Graph Instruction Tuning for Large Language Models
• Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning
• Learning on Large-scale Text-attributed Graphs via Variational Inference
• Balanced Multimodal Learning via On-the-fly Gradient Modulation
• Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)
• Is Homophily a Necessity for Graph Neural Networks?
• GraphSVX: Shapley Value Explanations for Graph Neural Networks
• On Explainability of Graph Neural Networks via Subgraph Explorations
• Extracting Training Data from Large Language Models
• Simple and Deep Graph Convolutional Networks
• Information-Theoretic Probing with Minimum Description Length
• Analysis Methods in Neural Language Processing: A Survey
• Pitfalls of Graph Neural Network Evaluation

Discussions

• Stop force-feeding LLM embeddings into GNNs. Research shows concatenation causes interference, degrading performance on homophilous graphs. For business, this means blind model integration creates tec [bsky, 1 points, 0 comments]

Related

• Graph Neural Networks are Inherently Good Generalizers: Insights by Bridging GNNs and MLPs
• Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation
• Is Graph Convolution Always Beneficial For Every Feature?
• Glance for Context: Learning When to Leverage LLMs for Node-Aware GNN-LLM Fusion
• Quantifying the Optimization and Generalization Advantages of Graph Neural Networks Over Multilayer Perceptrons