FuDoBa: Fusing Document and Knowledge Graph-based Representations with Bayesian Optimisation

2025/07/09 by Boshko Koloski, Senja Pollak, Koloski, Boshko +5
Computer Science · #Advanced Graph Neural Networks #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning in Healthcare #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2507.06622

openalex publication_date 2025/07/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

Building on the success of Large Language Models (LLMs), LLM-based representations have dominated the document representation landscape, achieving great performance on the document embedding benchmarks. However, the high-dimensional, computationally expensive embeddings from LLMs tend to be either too generic or inefficient for domain-specific applications. To address these limitations, we introduce FuDoBa a Bayesian optimisation-based method that integrates LLM-based embeddings with domain-specific structured knowledge, sourced both locally and from external repositories like WikiData. This fusion produces low-dimensional, task-relevant representations while reducing training complexity and yielding interpretable early-fusion weights for enhanced classification performance. We demonstrate the effectiveness of our approach on six datasets in two domains, showing that when paired with robust AutoML-based classifiers, our proposed representation learning approach performs on par with, or surpasses, those produced solely by the proprietary LLM-based embedding baselines.

Citations

• MAGNET: Augmenting Generative Decoders with Representation Learning and Infilling Capabilities
• Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering
• AutoML-guided Fusion of Entity and LLM-based Representations for Document Classification
• HybridRAG: Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extraction
• The Llama 3 Herd of Models
• Embedding-Aligned Language Models
• LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders
• Improving Text Embeddings with Large Language Models
• MTEB: Massive Text Embedding Benchmark
• The GINCO Training Dataset for Web Genre Identification of Documents Out in the Wild
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• Multilingual Offensive Language Identification with Cross-lingual\n Embeddings
• AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data
• KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation
• Using Local Knowledge Graph Construction to Scale Seq2Seq Models to Multi-Document Inputs
• Enriching BERT with Knowledge Graph Embeddings for Document Classification
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space
• Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects
• Distributed Representations of Sentences and Documents
• Where are your Manners? Sharing Best Community Practices in the Web 2.0
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• AutoML-guided Fusion of Entity and LLM-based Representations for Document Classification
• Robust Document Representations using Latent Topics and Metadata
• DocMamba: Efficient Document Pre-training with State Space Model
• SKILL: Structured Knowledge Infusion for Large Language Models
• Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training