ECG-LLM -- training and evaluation of domain-specific large language models for electrocardiography

2025/10/21 by Lara Ahrens, Ahrens, Lara, Wilhelm Haverkamp +3
Computer Science · Medicine · #Adaptation (eye) #Artificial Intelligence in Healthcare and Education #Base (topology) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Interpretability #Key (lock) #Knowledge base #Language model #Machine Learning (cs.LG) #Machine Learning in Healthcare #Ranking (information retrieval) #Software deployment #Task (project management) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2510.18339

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/10/21 · openalex created_date 2025/10/24 · openalex updated_date 2026/08/05

Abstract

Domain-adapted open-weight large language models (LLMs) offer promising healthcare applications, from queryable knowledge bases to multimodal assistants, with the crucial advantage of local deployment for privacy preservation. However, optimal adaptation strategies, evaluation methodologies, and performance relative to general-purpose LLMs remain poorly characterized. We investigated these questions in electrocardiography, an important area of cardiovascular medicine, by finetuning open-weight models on domain-specific literature and implementing a multi-layered evaluation framework comparing finetuned models, retrieval-augmented generation (RAG), and Claude Sonnet 3.7 as a representative general-purpose model. Finetuned Llama 3.1 70B achieved superior performance on multiple-choice evaluations and automatic text metrics, ranking second to Claude 3.7 in LLM-as-a-judge assessments. Human expert evaluation favored Claude 3.7 and RAG approaches for complex queries. Finetuned models significantly outperformed their base counterparts across nearly all evaluation modes. Our findings reveal substantial performance heterogeneity across evaluation methodologies, underscoring assessment complexity. Nevertheless, domain-specific adaptation through finetuning and RAG achieves competitive performance with proprietary models, supporting the viability of privacy-preserving, locally deployable clinical solutions.

Citations

• SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMs
• A multi-query, multimodal, receiver-augmented solution to extract contemporary cardiology guideline information using large language models
• CardAIc-Agents: A Multimodal Framework with Hierarchical Adaptation for Cardiac Care Support
• Can One Domain Help Others? A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learning
• MIRIAD: Augmenting LLMs with millions of medical query-response pairs
• Q-Heart: ECG Question Answering via Knowledge-Informed Multimodal LLMs
• Fine-Tuning LLMs on Small Medical Datasets: Text Classification and Normalization Effectiveness on Cardiology reports and Discharge records
• From Style to Facts: Mapping the Boundaries of Knowledge Injection with Finetuning
• RadPhi-3: Small Language Models for Radiology
• FineTuneBench: How well do commercial fine-tuning APIs infuse knowledge into LLMs?
• Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?
• How Much Can RAG Help the Reasoning of LLM?
• OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data
• Zodiac: A Cardiologist-Level LLM Framework for Multi-Agent Diagnostics
• MinerU: An Open-Source Solution for Precise Document Content Extraction
• ECG-Chat: A Large ECG-Language Model for Cardiac Disease Diagnosis
• From Text to Insight: Large Language Models for Materials Science Data Extraction
• From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
• Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models
• Enhancing Clinical Efficiency through LLM: Discharge Note Generation for Cardiac Patients
• Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
• Question answering systems for health professionals at the point of care -- a systematic review
• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
• InvestLM: A Large Language Model for Investment using Financial Domain Instruction Tuning
• A Comprehensive Overview of Large Language Models
• AlignScore: Evaluating Factual Consistency with a Unified Alignment Function
• Training language models to follow instructions with human feedback
• Measuring Massive Multitask Language Understanding
• Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• BERTScore: Evaluating Text Generation with BERT

Related

• ALFRED: Ask a Large-language model For Reliable ECG Diagnosis
• Retrieval-Augmented Generation for Electrocardiogram-Language Models
• De-biased Multimodal Electrocardiogram Analysis
• ECG-aBcDe: Overcoming Model Dependence, Encoding ECG into a Universal Language for Any LLM
• Transfer Knowledge from Natural Language to Electrocardiography: Can We Detect Cardiovascular Disease Through Language Models?