Parameter-Efficient Fine-Tuning of LLaMA for the Clinical Domain

2023/07/06 by Aryo Pradipta Gema, Gema, Aryo Pradipta, Pasquale Minervini +6 · 10 citations
Computer Science · Medicine · #Artificial Intelligence in Healthcare and Education #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning in Healthcare #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2307.03042

openalex publication_date 2023/07/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/03

Abstract

Adapting pretrained language models to novel domains, such as clinical applications, traditionally involves retraining their entire set of parameters. Parameter-Efficient Fine-Tuning (PEFT) techniques for fine-tuning language models significantly reduce computational requirements by selectively fine-tuning small subsets of parameters. In this study, we propose a two-step PEFT framework and evaluate it in the clinical domain. Our approach combines a specialised PEFT adapter layer designed for clinical domain adaptation with another adapter specialised for downstream tasks. We evaluate the framework on multiple clinical outcome prediction datasets, comparing it to clinically trained language models. Our framework achieves a better AUROC score averaged across all clinical downstream tasks compared to clinical language models. In particular, we observe large improvements of 4-5% AUROC in large-scale multilabel classification tasks, such as diagnoses and procedures classification. To our knowledge, this study is the first to provide an extensive empirical analysis of the interplay between PEFT techniques and domain adaptation in an important real-world domain of clinical applications.

Cited by

• Building Patient Journeys in Hebrew: A Language Model for Clinical Timeline Extraction
• OncoReason: Structuring Clinical Reasoning in LLMs for Robust and Interpretable Survival Prediction
• Evolution of meta's llama models and parameter-efficient fine-tuning of large language models: a survey
• CliniBench: A Clinical Outcome Prediction Benchmark for Generative and Encoder-Based Language Models
• HapticLLaMA: A Multimodal Sensory Language Model for Haptic Captioning
• What do Large Language Models know about materials?
• Benchmarking Foundation Models and Parameter-Efficient Fine-Tuning for Prognosis Prediction in Medical Imaging
• Aligned but Blind: Alignment Increases Implicit Bias by Reducing Awareness of Race
• Aligning Large Language Models with Healthcare Stakeholders: A Pathway to Trustworthy AI Integration
• Llama-3.1-FoundationAI-SecurityLLM-Base-8B Technical Report

Related

• Efficiency at Scale: Investigating the Performance of Diminutive Language Models in Clinical Tasks
• Can LLMs' Tuning Methods Work in Medical Multimodal Domain?
• Beyond Fine-tuning: Unleashing the Potential of Continuous Pretraining for Clinical LLMs
• Training-Free Adaptation of New-Generation LLMs using Legacy Clinical Models
• Evolution of meta's llama models and parameter-efficient fine-tuning of large language models: a survey