Publicly Available Clinical BERT Embeddings

2019/04/06 by Emily Alsentzer, John R. Murphy, Alsentzer, Emily +11 · 104 citations
Computer Science · #Topic Modeling #Natural Language Processing Techniques #Text Readability and Simplification

paper · pdf · doi:10.48550/arxiv.1904.03323

Abstract

Contextual word embedding models such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2018) have dramatically improved performance for many natural language processing (NLP) tasks in recent months. However, these models have been minimally explored on specialty corpora, such as clinical text; moreover, in the clinical domain, no publicly-available pre-trained BERT models yet exist. In this work, we address this need by exploring and releasing BERT models for clinical text: one for generic clinical text and another for discharge summaries specifically. We demonstrate that using a domain-specific model yields performance improvements on three common clinical NLP tasks as compared to nonspecific embeddings. These domain-specific models are not as performant on two clinical de-identification tasks, and argue that this is a natural consequence of the differences between de-identified source text and synthetically non de-identified task text.

Cited by

• Spectral Dynamics of Semantic Drift in Clinical Multi-Agent Language Model Networks
• Training-Free Adaptation of New-Generation LLMs using Legacy Clinical Models
• Beyond CLIP: Knowledge-Enhanced Multimodal Transformers for Cross-Modal Alignment in Diabetic Retinopathy Diagnosis
• HARMON-E: Hierarchical Agentic Reasoning for Multimodal Oncology Notes to Extract Structured Data
• Early Warning Index for Patient Deteriorations in Hospitals
• LLM-driven Knowledge Enhancement for Multimodal Cancer Survival Prediction
• SepsisSuite: Beyond Risk Stratification -- A Comparative Analysis of Deep Fusion vs. Expert Stacking for Prescriptive Sepsis AI
• Automated Identification of Incidentalomas Requiring Follow-Up: A Multi-Anatomy Evaluation of LLM-Based and Supervised Approaches
• Text Rationalization for Robust Causal Effect Estimation
• SP-Det: Self-Prompted Dual-Text Fusion for Generalized Multi-Label Lesion Detection
• COPE: Chain-Of-Thought Prediction Engine for Open-Source Large Language Model Based Stroke Outcome Prediction from Clinical Notes
• Explainable Health Risk Predictor with Transformer-based Medicare Claim Encoder
• A Fast and Efficient Modern BERT based Text-Conditioned Diffusion Model for Medical Image Segmentation
• A Systematic Analysis of Large Language Models with RAG-enabled Dynamic Prompting for Medical Error Detection and Correction
• Comparative Analysis of LoRA-Adapted Embedding Models for Clinical Cardiology Text Representation
• Boosting Medical Visual Understanding From Multi-Granular Language Learning
• OEMA: Ontology-Enhanced Multi-Agent Collaboration Framework for Zero-Shot Clinical Named Entity Recognition
• Transparent Early ICU Mortality Prediction with Clinical Transformer and Per-Case Modality Attribution
• Systematic Reconstruction of Disease Networks from Longitudinal Blood Data for Causal Discovery and Intervention Analysis
• H-LDM: Hierarchical Latent Diffusion Models for Controllable and Interpretable PCG Synthesis from Clinical Metadata
• Zero-Training Task-Specific Model Synthesis for Few-Shot Medical Image Classification
• Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction
• A Disease-Aware Dual-Stage Framework for Chest X-ray Report Generation
• SurvBench: A Standardised Preprocessing Pipeline for Multi-Modal Electronic Health Record Survival Analysis
• A Meta-Evaluation of Faithfulness Metrics for Long-Form Hospital-Course Summarization
• SCALE-VLP: Soft-Weighted Contrastive Volumetric Vision-Language Pre-training with Spatial-Knowledge Semantics
• Count-Based Approaches Remain Strong: A Benchmark Against Transformer and LLM Pipelines on Structured EHR
• multiMentalRoBERTa: A Fine-tuned Multiclass Classifier for Mental Health Disorder
• MedM2T: A MultiModal Framework for Time-Aware Modeling with Electronic Health Record and Electrocardiogram Data
• Overview of the MEDIQA-OE 2025 Shared Task on Medical Order Extraction from Doctor-Patient Consultations
• MV-MLM: Bridging Multi-View Mammography and Language for Breast Cancer Diagnosis and Risk Prediction
• Quantifying Explainability in NLP and Analyzing Algorithms for Performance-Explainability Tradeoff
• Recent Advances in Named Entity Recognition: A Comprehensive Survey and Comparative Study
• MedFusionT5: Cross-Modal Attention Boosts Semantic Quality and Reduces Hallucinations in Dental AI
• Breast Cancer VLMs: Clinically Practical Vision-Language Train-Inference Models
• When are radiology reports useful for training medical image classifiers?
• CLIN-LLM: A Safety-Constrained Hybrid Framework for Clinical Diagnosis and Treatment Generation
• PSScreen V2: Partially Supervised Multiple Retinal Disease Screening
• Supervised Fine-Tuning or In-Context Learning? Evaluating LLMs for Clinical NER
• Automated Extraction of Fluoropyrimidine Treatment and Treatment-Related Toxicities from Clinical Notes Using Natural Language Processing
• Investigating LLM Capabilities on Long Context Comprehension for Medical Question Answering
• Med-VRAgent: A Framework for Medical Visual Reasoning-Enhanced Agents
• Enhancing mortality prediction in cardiac arrest ICU patients through meta-modeling of structured clinical data from MIMIC-IV
• Multilingual Clinical NER for Diseases and Medications Recognition in Cardiology Texts using BERT Embeddings
• Renaissance of RNNs in Streaming Clinical Time Series: Compact Recurrence Remains Competitive with Transformers
• MOSAIC: Masked Objective with Selective Adaptation for In-domain Contrastive Learning
• TACL: Threshold-Adaptive Curriculum Learning Strategy for Enhancing Medical Text Understanding
• JEDA: Query-Free Clinical Order Search from Ambient Dialogues
• Automated Labelling using an Attention model for Radiology reports of\n MRI scans (ALARM)
• GapDNER: A Gap-Aware Grid Tagging Model for Discontinuous Named Entity Recognition
• Assessing Large Language Models for Structured Medical Order Extraction
• Lightweight Baselines for Medical Abstract Classification: DistilBERT with Cross-Entropy as a Strong Default
• Serialized EHR make for good text representations
• Evaluating Fundus-Specific Foundation Models for Diabetic Macular Edema Detection
• KEEP: Integrating Medical Ontologies with Clinical Data for Robust Code Embeddings
• Resource-Efficient Fine-Tuning of LLaMA-3.2-3B for Medical Chain-of-Thought Reasoning
• Building the EHR Foundation Model via Next Event Prediction
• AutoPK: Leveraging LLMs and a Hybrid Similarity Metric for Advanced Retrieval of Pharmacokinetic Data from Complex Tables and Documents
• Mammo-CLIP Dissect: A Framework for Analysing Mammography Concepts in Vision-Language Models
• Revolutionizing Precise Low Back Pain Diagnosis via Contrastive Learning
• RAD: Towards Trustworthy Retrieval-Augmented Multi-modal Clinical Diagnosis
• Biomedical Entity Representations with Synonym Marginalization
• Improving Clinical Outcome Predictions Using Convolution over Medical Entities with Multimodal Learning
• Systematic Comparative Analysis of Large Pretrained Language Models on Contextualized Medication Event Extraction
• Unified Multimodal Coherent Field: Synchronous Semantic-Spatial-Vision Fusion for Brain Tumor Segmentation
• Causal Representation Learning from Multimodal Clinical Records under Non-Random Modality Missingness
• MedCutMix: A Data-Centric Approach to Improve Radiology Vision-Language Pre-training with Disease Awareness
• Scientific Language Models for Biomedical Knowledge Base Completion: An Empirical Study
• Not What the Doctor Ordered: Surveying LLM-based De-identification and Quantifying Clinical Information Loss
• Exploring the Capabilities of LLM Encoders for Image-Text Retrieval in Chest X-rays
• Integrating Text and Time-Series into (Large) Language Models to Predict Medical Outcomes
• RanAT4BIE: Random Adversarial Training for Biomedical Information Extraction
• GLAM: Geometry-Guided Local Alignment for Multi-View VLP in Mammography
• BALI: Enhancing Biomedical Language Representations through Knowledge Graph and Language Model Alignment
• Classification of kinetic-related injury in hospital triage data using NLP
• Semantic Analysis of SNOMED CT Concept Co-occurrences in Clinical Documentation using MIMIC-IV
• Generative Foundation Model for Structured and Unstructured Electronic Health Records
• A Hybrid AI-based and Rule-based Approach to DICOM De-identification: A Solution for the MIDI-B Challenge
• Benchmarking GPT-5 in Radiation Oncology: Measurable Gains, but Persistent Need for Expert Oversight
• Ontology-Based Concept Distillation for Radiology Report Retrieval and Labeling
• MedVQA-TREE: A Multimodal Reasoning and Retrieval Framework for Sarcopenia Prediction
• Subjective Behaviors and Preferences in LLM: Language of Browsing
• Coarse-to-Fine Personalized LLM Impressions for Streamlined Radiology Reports
• A Fully Transformer Based Multimodal Framework for Explainable Cancer Image Segmentation Using Radiology Reports
• A Language-Signal-Vision Multimodal Framework for Multitask Cardiac Analysis
• Eyes on the Image: Gaze Supervised Multimodal Learning for Chest X-ray Diagnosis and Report Generation
• CardAIc-Agents: A Multimodal Framework with Hierarchical Adaptation for Cardiac Care Support
• Benchmarking the Medical Understanding and Reasoning of Large Language Models in Arabic Healthcare Tasks
• P-CAFE: Personalized Cost-Aware Incremental Feature Selection For Electronic Health Records
• Leveraging Large Language Models for Rare Disease Named Entity Recognition
• FEAT: A Multi-Agent Forensic AI System with Domain-Adapted Large Language Model for Automated Cause-of-Death Analysis
• MAQuA: Adaptive Question-Asking for Multidimensional Mental Health Screening using Item Response Theory
• LLMCARE: early detection of cognitive impairment via transformer models enhanced by LLM-generated synthetic data
• MoMA: A Mixture-of-Multimodal-Agents Architecture for Enhancing Clinical Prediction Modelling
• Large Language Model's Multi-Capability Alignment in Biomedical Domain
• R2GenKG: Hierarchical Multi-modal Knowledge Graph for LLM-based Radiology Report Generation
• OpenMed NER: Open-Source, Domain-Adapted State-of-the-Art Transformers for Biomedical NER Across 12 Public Datasets
• Read, Attend, and Code: Pushing the Limits of Medical Codes Prediction from Clinical Notes by Machines
• Distribution-Based Masked Medical Vision-Language Model Using Structured Reports
• On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey
• Phenotyping of Clinical Notes with Improved Document Classification Models Using Contextualized Neural Language Models
• Bridging AI Innovation and Healthcare Needs: Lessons Learned from Incorporating Modern NLP at The BC Cancer Registry
• AutoPCR: Automated Phenotype Concept Recognition by Prompting
• #StayHome or #Marathon? Social Media Enhanced Pandemic Surveillance on Spatial-temporal Dynamic Graphs

Related

• Federated pretraining and fine tuning of BERT using clinical notes from multiple silos
• MedicalBERT: enhancing biomedical natural language processing using pretrained BERT-based model
• Probing Biomedical Embeddings from Language Models
• Generalist embedding models are better at short-context clinical semantic search than specialized embedding models
• How Can BERT Help Lexical Semantics Tasks?