Mixture-of-Partitions: Infusing Large Biomedical Knowledge Graphs into BERT

2021/09/10 by Zaiqiao Meng, Meng, Zaiqiao, Fangyu Liu +8 · 1 citation
Computer Science · #Advanced Graph Neural Networks #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning in Healthcare #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.2109.04810

EMNLP 2021 camera-ready version

arxiv created 2021/09/10 · openalex publication_date 2021/09/10 · arxiv updated 2021/09/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Infusing factual knowledge into pre-trained models is fundamental for many knowledge-intensive tasks. In this paper, we proposed Mixture-of-Partitions (MoP), an infusion approach that can handle a very large knowledge graph (KG) by partitioning it into smaller sub-graphs and infusing their specific knowledge into various BERT models using lightweight adapters. To leverage the overall factual knowledge for a target task, these sub-graph adapters are further fine-tuned along with the underlying BERT through a mixture layer. We evaluate our MoP with three biomedical BERTs (SciBERT, BioBERT, PubmedBERT) on six downstream tasks (inc. NLI, QA, Classification), and the results show that our MoP consistently enhances the underlying BERTs in task performance, and achieves new SOTA performances on five evaluated datasets.

Citations

• Distilling the Knowledge in a Neural Network
• A Fast and High Quality Multilevel Scheme for Partitioning Irregular Graphs
• Distributed Representations of Words and Phrases and their Compositionality
• Categorical Reparameterization with Gumbel-Softmax
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Decoupled Weight Decay Regularization
• ERNIE: Enhanced Representation through Knowledge Integration
• Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model
• K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
• DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphs

Cited by

• Triplet-Structured Knowledge Integration for Multi-Turn Medical Reasoning

Related

• Contrastive Learning and Mixture of Experts Enables Precise Vector Embeddings
• KG-MTT-BERT: Knowledge Graph Enhanced BERT for Multi-Type Medical Text Classification
• Diversifying Knowledge Enhancement of Biomedical Language Models using Adapter Modules and Knowledge Graphs
• Federated pretraining and fine tuning of BERT using clinical notes from multiple silos
• BioGraphFusion: Graph Knowledge Embedding for Biological Completion and Reasoning