ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
2019/04/10 by Kexin Huang, Huang, Kexin, Jaan Altosaar +3 · 137 citations
Computer Science · Health Professions · #Computation and Language (cs.CL) #Electronic Health Records Systems #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning in Healthcare #Topic Modeling #cs.CL #cs.LG
paper · pdf · doi:10.48550/arxiv.1904.05342
CHIL 2020 Workshop
openalex publication_date 2019/04/10 · openalex created_date 2019/04/25 · arxiv created 2020/11/29 · arxiv updated 2020/12/01 · openalex updated_date 2026/07/28
Abstract
Clinical notes contain information about patients that goes beyond structured data like lab values and medications. However, clinical notes have been underused relative to structured data, because notes are high-dimensional and sparse. This work develops and evaluates representations of clinical notes using bidirectional transformers (ClinicalBERT). ClinicalBERT uncovers high-quality relationships between medical concepts as judged by humans. ClinicalBert outperforms baselines on 30-day hospital readmission prediction using both discharge summaries and the first few days of notes in the intensive care unit. Code and model parameters are available.
Citations
Cited by
- Closed-Loop Validation-Repair for Healthcare Interoperability: A Multi-Model Study of Schema Compliance in Clinical LLMs
- OrganLens: Organ-Specific Representation Learning for CT Foundation Models
- Medical world models in healthcare: foundations, applications, and challenges for trustworthy clinical translation
- Aletheia: An Offline-First Clinical Decision Support System for Differential Diagnosis in Low-Resource Healthcare Settings
- Evaluating large language models for diagnostic reasoning from unstructured clinical narratives in epilepsy
- Fixed-Budget Parameter-Efficient Training with Frozen Encoders Improves Multimodal Chest X-Ray Classification
- Multi-Agent Medical Decision Consensus Matrix System: An Intelligent Collaborative Framework for Oncology MDT Consultations
- A Practical Framework for Evaluating Medical AI Security: Reproducible Assessment of Jailbreaking and Privacy Vulnerabilities Across Clinical Specialties
- ClinNoteAgents: An LLM Multi-Agent System for Predicting and Interpreting Heart Failure 30-Day Readmission from Clinical Notes
- Automated Identification of Incidentalomas Requiring Follow-Up: A Multi-Anatomy Evaluation of LLM-Based and Supervised Approaches
- Text Rationalization for Robust Causal Effect Estimation
- COPE: Chain-Of-Thought Prediction Engine for Open-Source Large Language Model Based Stroke Outcome Prediction from Clinical Notes
- PULSE-ICU: A Pretrained Unified Long-Sequence Encoder for Multi-task Prediction in Intensive Care Units
- OEMA: Ontology-Enhanced Multi-Agent Collaboration Framework for Zero-Shot Clinical Named Entity Recognition
- QwenCLIP: Boosting Medical Vision-Language Pretraining via LLM Embeddings and Prompt tuning
- MedPath: Multi-Domain Cross-Vocabulary Hierarchical Paths for Biomedical Entity Linking
- ClinStructor: AI-Powered Structuring of Unstructured Clinical Texts
- SurvBench: A Standardised Preprocessing Pipeline for Multi-Modal Electronic Health Record Survival Analysis
- The Open Syndrome Definition as a Machine-Readable Standard for Public Health: Design and Implementation Study
- Fine-Tuning DialoGPT on Common Diseases in Rural Nepal for Medical Conversations
- Multilingual BERT language model for medical tasks: Evaluation on domain-specific adaptation and cross-linguality
- A Memory-Efficient Retrieval Architecture for RAG-Enabled Wearable Medical LLMs-Agents
- Enhancing Biomedical Named Entity Recognition using GLiNER-BioMed with Targeted Dictionary-Based Post-processing for BioASQ 2025 task 6
- Improving Heart-Focused Medical Question Answering in LLMs via Variance-Aware Rubric Rewards with GRPO
- Predicting life satisfaction using machine learning and explainable AI
- Generative AI for Healthcare: Fundamentals, Challenges, and Perspectives
- When are radiology reports useful for training medical image classifiers?
- Leveraging Hierarchical Organization for Medical Multi-document Summarization
- Leverage Unlearning to Sanitize LLMs
- FinBERT: A Pretrained Language Model for Financial Communications
- Exploring Membership Inference Vulnerabilities in Clinical Large Language Models
- Large language models in medicine
- Enhancing mortality prediction in cardiac arrest ICU patients through meta-modeling of structured clinical data from MIMIC-IV
- TACL: Threshold-Adaptive Curriculum Learning Strategy for Enhancing Medical Text Understanding
- JEDA: Query-Free Clinical Order Search from Ambient Dialogues
- Towards Human-Centric Intelligent Treatment Planning for Radiation Therapy
- Serialized EHR make for good text representations
- SMedBERT: A Knowledge-Enhanced Pre-trained Language Model with Structured Semantics for Medical Text Mining
- Alignment, Mining and Fusion: Representation Alignment with Hard Negative Mining and Selective Knowledge Fusion for Medical Visual Question Answering
- Toward Reliable Clinical Coding with Language Models: Verification and Lightweight Adaptation
- Revealing Interconnections between Diseases: from Statistical Methods to Large Language Models
- Contrastive Learning Using Graph Embeddings for Domain Adaptation of Language Models in the Process Industry
- PRISM-Consult: A Panel-of-Experts Architecture for Clinician-Aligned Diagnosis
- Retrieval-Augmented Framework for LLM-Based Clinical Decision Support
- Darling (v2.0): Mining disease-related databases for the detection of biomedical entity associations
- Expert-guided Clinical Text Augmentation via Query-Based Model Collaboration
- Revisiting Performance Claims for Chest X-Ray Models Using Clinical Context
- The MADRS Pipeline: Supporting Depression Assessment in Clinical Trials
- Improving Clinical Outcome Predictions Using Convolution over Medical Entities with Multimodal Learning
- ReDepress: A Cognitive Framework for Detecting Depression Relapse from Social Media
- Causal Representation Learning from Multimodal Clinical Records under Non-Random Modality Missingness
- Uncertainty Quantification of Large Language Models using Approximate Bayesian Computation
- Efficient and Versatile Model for Multilingual Information Retrieval of Islamic Text: Development and Deployment in Real-World Scenarios
- Radiology Report Conditional 3D CT Generation with Multi Encoder Latent diffusion Model
- Not What the Doctor Ordered: Surveying LLM-based De-identification and Quantifying Clinical Information Loss
- Fast, Structured Clinical Documentation via Contextual Autocomplete
- On the Universality of Deep Contextual Language Models
- Transformer Query-Target Knowledge Discovery (TEND): Drug Discovery from CORD-19
- Enriched text-guided variational multimodal knowledge distillation network (VMD) for automated diagnosis of plaque vulnerability in 3D carotid artery MRI
- BioMegatron: Larger Biomedical Domain Language Model
- MS2: Multi-Document Summarization of Medical Studies
- Domain-Aware RAG: MoL-Enhanced RL for Efficient Training and Scalable Retrieval
- A Study of Large Language Models for Patient Information Extraction: Model Architecture, Fine-Tuning Strategy, and Multi-task Instruction Tuning
- Hierarchical Section Matching Prediction (HSMP) BERT for Fine-Grained Extraction of Structured Data from Hebrew Free-Text Radiology Reports in Crohn's Disease
- Enriched Annotations for Tumor Attribute Classification from Pathology Reports with Limited Labeled Data
- Weakly Supervised Medical Entity Extraction and Linking for Chief Complaints
- Efficient Domain Adaptation of Language Models via Adaptive Tokenization
- Automated Quality Assessment for LLM-Based Complex Qualitative Coding: A Confidence-Diversity Framework
- A Robust BERT-Based Deep Learning Model for Automated Cancer Type Extraction from Unstructured Pathology Reports
- A Laplace diffusion-based transformer model for heart rate forecasting within daily activity context
- Organ-Agents: Virtual Human Physiology Simulator via LLMs
- Extracting Post-Acute Sequelae of SARS-CoV-2 Infection Symptoms from Clinical Notes via Hybrid Natural Language Processing
- Bi-Axial Transformers: Addressing the Increasing Complexity of EHR Classification
- QuarkMed Medical Foundation Model Technical Report
- AI in Mental Health: Emotional and Sentiment Analysis of Large Language Models' Responses to Depression, Anxiety, and Stress Queries
- Memory Decoder: A Pretrained, Plug-and-Play Memory for Large Language Models
- Predicting Unplanned Readmissions with Highly Unstructured Data
- Skin-SOAP: A Weakly Supervised Framework for Generating Structured SOAP Notes
- RooseBERT: A New Deal For Political Language Modelling
- Clinical XLNet: Modeling Sequential Clinical Notes and Predicting Prolonged Mechanical Ventilation
- From EMR Data to Clinical Insight: An LLM-Driven Framework for Automated Pre-Consultation Questionnaire Generation
- Deep Learning-based Prediction of Clinical Trial Enrollment with Uncertainty Estimates
- How Far Are AI Scientists from Changing the World?
- Advances in Multi-turn Dialogue Comprehension: A Survey
- Towards Domain Specification of Embedding Models in Medicine
- Improving Dialogue Breakdown Detection with Semi-Supervised Learning
- DENSE: Longitudinal Progress Note Generation with Temporal Modeling of Heterogeneous Clinical Notes Across Hospital Visits
- PubMind: Literature-Based Genetic Variant Extraction and Functional Annotation Using Large Language Models
- CU-ICU: Customizing Unsupervised Instruction-Finetuned Language Models for ICU Datasets via Text-to-Text Transfer Transformer
- Pre-trained Language Models in Biomedical Domain: A Systematic Survey
- Pre-training Language Model Incorporating Domain-specific Heterogeneous Knowledge into A Unified Representation
- Machine Learning Applications for Therapeutic Tasks with Genomics Data
- Enhancing Clinical Text Classification via Fine-Tuned DRAGON Longformer Models
- Multi-modal Representations for Fine-grained Multi-label Critical View of Safety Recognition
- Cost-effective Selection of Pretraining Data: A Case Study of Pretraining BERT on Social Media
- MT-BioNER: Multi-task Learning for Biomedical Named Entity Recognition using Deep Bidirectional Transformers
- LLM-Driven Treatment Effect Estimation Under Inference Time Text Confounding
- Large Language Models for Automating Clinical Data Standardization: HL7 FHIR Use Case
- Improving Early Sepsis Prediction with Multi Modal Learning
- Domain-Relevant Embeddings for Medical Question Similarity
- Holistic Artificial Intelligence in Medicine; improved performance and explainability
- Position: Use Sparse Autoencoders to Discover Unknowns
- Federated Timeline Synthesis: Scalable and Private Methodology For Model Training and Deployment
- Prompt Mechanisms in Medical Imaging: A Comprehensive Survey
- Clinical Relation Extraction Using Transformer-based Models
- Cohort Retrieval using Dense Passage Retrieval
- Refine Medical Diagnosis Using Generation Augmented Retrieval and Clinical Practice Guidelines
- THCM-CAL: Temporal-Hierarchical Causal Modelling with Conformal Calibration for Clinical Risk Prediction
- Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Media
- CKD-EHR:Clinical Knowledge Distillation for Electronic Health Records
- Discovering Hierarchy-Grounded Domains with Adaptive Granularity for Clinical Domain Generalization
- Leveraging MIMIC Datasets for Better Digital Health: A Review on Open Problems, Progress Highlights, and Future Promises
- Recent Advances and Future Directions in Literature-Based Discovery
- Generative or Discriminative? Revisiting Text Classification in the Era of Transformers
- MedCite: Can Language Models Generate Verifiable Text for Medicine?
- Tau-Eval: A Unified Evaluation Framework for Useful and Private Text Anonymization
- Constrained Entropic Unlearning: A Primal-Dual Framework for Large Language Models
- A Brain Graph Foundation Model: Pre-Training and Prompt-Tuning across Broad Atlases and Disorders
- A Large Language Model Based Pipeline for Review of Systems Entity Recognition from Clinical Notes
- A General Knowledge Injection Framework for ICD Coding
- Representation Learning for Electronic Health Records
- Don't be fooled: label leakage in explanation methods and the importance of their quantitative evaluation
- Hurtful Words: Quantifying Biases in Clinical Contextual Word Embeddings
- KDH-MLTC: Knowledge Distillation for Healthcare Multi-Label Text Classification
- Train No Evil: Selective Masking for Task-Guided Pre-Training
- Aligning Large Language Models with Healthcare Stakeholders: A Pathway to Trustworthy AI Integration
- Application of Pre-training Models in Named Entity Recognition
- Enhancing Security and Strengthening Defenses in Automated Short-Answer Grading Systems
- REBENCH: A Procedural, Fair-by-Construction Benchmark for LLMs on Stripped-Binary Types and Names (Extended Version)
- Hallucinations and Key Information Extraction in Medical Texts: A Comprehensive Assessment of Open-Source Large Language Models
- Temporal Entailment Pretraining for Clinical Language Models over EHR Data
- HMI: Hierarchical Knowledge Management for Efficient Multi-Tenant Inference in Pretrained Language Models
- Patients-like-me: A Variational LM--GNN Framework for Explainable Clinical Prediction
- CSS-LM: A Contrastive Framework for Semi-Supervised Fine-Tuning of Pre-Trained Language Models
- Artificial intelligence in preclinical research: enhancing digital twins and organ-on-chip to reduce animal testing
- A Scoping Review of Natural Language Processing in Addressing Medically Inaccurate Information: Errors, Misinformation, and Hallucination
- Can We Edit LLMs for Long-Tail Biomedical Knowledge?
Related