Contrastive Learning Using Graph Embeddings for Domain Adaptation of Language Models in the Process Industry

2025/10/06 by Anastasia Zhukova, Zhukova, Anastasia, Jonas Lührs +5
Computer Science · #Advanced Graph Neural Networks #Computation and Language (cs.CL) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2510.04631

openalex publication_date 2025/10/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent trends in NLP utilize knowledge graphs (KGs) to enhance pretrained language models by incorporating additional knowledge from the graph structures to learn domain-specific terminology or relationships between documents that might otherwise be overlooked. This paper explores how SciNCL, a graph-aware neighborhood contrastive learning methodology originally designed for scientific publications, can be applied to the process industry domain, where text logs contain crucial information about daily operations and are often structured as sparse KGs. Our experiments demonstrate that language models fine-tuned with triplets derived from graph embeddings (GE) outperform a state-of-the-art mE5-large text encoder by 9.8-14.3% (5.45-7.96p) on the proprietary process industry text embedding benchmark (PITEB) while having 3 times fewer parameters.

Citations

• Link Prediction for Event Logs in the Process Industry
• Efficient Domain-adaptive Continual Pretraining for the Process Industry in the German Language
• Training Sparse Mixture Of Experts Text Embedding Models
• Automated Collection of Evaluation Dataset for Semantic Search in\n Low-Resource Domain Language
• mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval
• Multilingual E5 Text Embeddings: A Technical Report
• M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
• Generative User-Experience Research for Developing Domain-specific Natural Language Processing Applications
• TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations at Twitter
• LinkBERT: Pretraining Language Models with Document Links
• Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings
• BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models
• German's Next Language Model
• MaintNet: A Collaborative Open-Source Library for Predictive Maintenance Language Resources
• Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
• Don't Stop Pretraining: Adapt Language Models to Domains and Tasks
• SPECTER: Document-level Representation Learning using Citation-informed Transformers
• KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation
• Unsupervised Cross-lingual Representation Learning at Scale
• K-BERT: Enabling Language Representation with Knowledge Graph
• Knowledge Enhanced Contextual Word Representations
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• ERNIE: Enhanced Language Representation with Informative Entities
• ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
• A Theoretical Analysis of Contrastive Unsupervised Representation Learning
• Sampling Matters in Deep Embedding Learning
• Billion-scale similarity search with GPUs
• MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• LATEX-GCL: Large Language Models (LLMs)-Based Data Augmentation for Text-Attributed Graph Contrastive Learning
• Towards Text-free Graph Foundation Models: Rethinking Multi-Domain Graph Contrastive Learning
• Graph Language Models
• SE-GCL: An Event-Based Simple and Effective Graph Contrastive Learning for Text Representation
• GT2Vec: Large Language Models as Multi-Modal Encoders for Text and Graph-Structured Data