Large Language Models are Powerful Electronic Health Record Encoders

2025/02/24 by Stefan Hegselmann, Georg von Arnim, Hegselmann, Stefan +13 · 1 voice · 9 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #cs.AI #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2502.17403

arxiv published 2025/02/24 · arxiv updated 2026/04/13

Abstract

Electronic Health Records (EHRs) offer considerable potential for clinical prediction, but their complexity and heterogeneity challenge traditional machine learning. Domain-specific EHR foundation models trained on unlabeled EHR data have shown improved predictive accuracy and generalization. However, their development is constrained by limited data access and site-specific vocabularies. We convert EHR data into plain text by replacing medical codes with natural-language descriptions, enabling general-purpose Large Language Models (LLMs) to produce high-dimensional embeddings for downstream prediction tasks without access to private medical training data. LLM-based embeddings perform on par with a specialized EHR foundation model, CLMBR-T-Base, across 15 clinical tasks from the EHRSHOT benchmark. In an external validation using the UK Biobank, an LLM-based model shows statistically significant improvements for some tasks, which we attribute to higher vocabulary coverage and slightly better generalization. Overall, we reveal a trade-off between the computational efficiency of specialized EHR models and the portability and data independence of LLM-based embeddings.

Cited by

• Fusion or Confusion? Multimodal Complexity Is Not All You Need
• Rethinking Tokenization for Clinical Time Series: When Less is More
• Aligning Probabilistic Beliefs under Informative Missingness: LLM Steerability in Clinical Reasoning
• APRIL: Annotations for Policy evaluation with Reliable Inference from LLMs
• EHR-R1: A Reasoning-Enhanced Foundational Language Model for Electronic Health Record Analysis
• The CRITICAL Records Integrated Standardization Pipeline (CRISP): End-to-End Processing of Large-scale Multi-institutional OMOP CDM Data
• LLMs can construct powerful representations and streamline sample-efficient supervised learning
• Patients-like-me: A Variational LM--GNN Framework for Explainable Clinical Prediction
• ChronoFormer: Time-Aware Transformer Architectures for Structured Clinical Event Modeling

Discussions

• Via Ian Maurer on LinkedIn: “Large Language Models are Powerful EHR Encoders”. Promising results for informatics applications. arxiv.org/abs/2502.17403 [bsky, 1 points, 0 comments]

Related

• Prompting Large Language Models for Zero-Shot Clinical Prediction with Structured Longitudinal Electronic Health Record Data
• A Comprehensive Survey of Electronic Health Record Modeling: From Deep Learning Approaches to Large Language Models
• Language Models Are An Effective Patient Representation Learning Technique For Electronic Health Record Data
• Will Large Language Models Transform Clinical Prediction?
• Clinical Risk Prediction Using Language Models: Benefits And Considerations