KG-BiLM: Knowledge Graph Embedding via Bidirectional Language Models

2025/06/04 by Zirui Chen, Chen, Zirui, Xin Wang +7
Computer Science · #Advanced Graph Neural Networks #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning in Healthcare #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2506.03576

openalex publication_date 2025/06/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent advances in knowledge representation learning (KRL) highlight the urgent necessity to unify symbolic knowledge graphs (KGs) with language models (LMs) for richer semantic understanding. However, existing approaches typically prioritize either graph structure or textual semantics, leaving a gap: a unified framework that simultaneously captures global KG connectivity, nuanced linguistic context, and discriminative reasoning semantics. To bridge this gap, we introduce KG-BiLM, a bidirectional LM framework that fuses structural cues from KGs with the semantic expressiveness of generative transformers. KG-BiLM incorporates three key components: (i) Bidirectional Knowledge Attention, which removes the causal mask to enable full interaction among all tokens and entities; (ii) Knowledge-Masked Prediction, which encourages the model to leverage both local semantic contexts and global graph connectivity; and (iii) Contrastive Graph Semantic Aggregation, which preserves KG structure via contrastive alignment of sampled sub-graph representations. Extensive experiments on standard benchmarks demonstrate that KG-BiLM outperforms strong baselines in link prediction, especially on large-scale graphs with complex multi-hop relations - validating its effectiveness in unifying structural information and textual semantics.

Citations

• MKGL: Mastery of a Three-Word Language
• KG-FIT: Knowledge Graph Fine-Tuning Upon Open-World Knowledge
• Advancing Graph Representation Learning with Large Language Models: A Comprehensive Survey of Techniques
• Contextualization Distillation from Large Language Model for Knowledge Graph Completion
• Enhancing Text-based Knowledge Graph Completion with Zero-Shot Large Language Models: A Focus on Semantic Enhancement
• Making Large Language Models Perform Better in Knowledge Graph Completion
• Exploring Large Language Models for Knowledge Graph Completion
• Large Language Models and Knowledge Graphs: Opportunities and Challenges
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Dipping PLMs Sauce: Bridging Structure and Text for Effective Knowledge Graph Completion via Conditional Soft Prompting
• Unifying Large Language Models and Knowledge Graphs: A Roadmap
• Enhancing Knowledge Graph Construction Using Large Language Models
• GPT-4 Technical Report
• LLaMA: Open and Efficient Foundation Language Models
• Joint Language Semantic and Structure Embedding for Knowledge Graph Completion
• Knowledge Is Flat: A Seq2Seq Generative Framework for Various Knowledge Graph Completion
• Language Models as Knowledge Embeddings
• ExpressivE: A Spatio-Functional Embedding For Knowledge Graph Completion
• Sequence-to-Sequence Knowledge Graph Completion and Question Answering
• SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Models
• Reasoning Through Memorization: Nearest Neighbor Knowledge Graph Embeddings
• Multi-task Pre-training Language Model for Semantic Network Completion
• Improving Knowledge Graph Representation Learning by Structure Contextual Pre-training
• Modeling Heterogeneous Hierarchies with Relation-specific Hyperbolic Cones
• Rot-Pro: Modeling Transitivity by Projection in Knowledge Graph Embedding
• Neural Bellman-Ford Networks: A General Graph Neural Network Framework for Link Prediction
• Robust Knowledge Graph Completion with Stacked Convolutions and a Student Re-Ranking Network
• QuatDE: Dynamic Quaternion Embedding for Knowledge Graph Completion
• Duality-Induced Regularizer for Tensor Factorization Based Knowledge Graph Completion
• MLMLM: Link Prediction with Mean Likelihood Masked Language Model
• Language Models are Few-Shot Learners
• A Survey on Knowledge Graphs: Representation, Acquisition, and Applications
• KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation
• Composition-based Multi-Relational Graph Convolutional Networks
• K-BERT: Enabling Language Representation with Knowledge Graph
• KG-BERT: BERT for Knowledge Graph Completion
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• Quaternion Knowledge Graph Embeddings
• RotatE: Knowledge Graph Embedding by Relational Rotation in Complex\n Space
• Representation Learning with Contrastive Predictive Coding
• KBGAN: Adversarial Learning for Knowledge Graph Embeddings
• Convolutional 2D Knowledge Graph Embeddings
• Modeling Relational Data with Graph Convolutional Networks
• Knowledge Graph Representation with Jointly Structural and Textual Encoding
• Complex Embeddings for Simple Link Prediction
• Holographic Embeddings of Knowledge Graphs
• Embedding Entities and Relations for Learning and Inference in Knowledge Bases
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• Injecting Knowledge Graphs into Large Language Models
• KGLM: Integrating Knowledge Graph Structure in Language Models for Link Prediction
• KG-FIT: Knowledge Graph Fine-Tuning Upon Open-World Knowledge
• LLM-empowered knowledge graph construction: A survey
• SEMMA: A Semantic Aware Knowledge Graph Foundation Model