Domain-Specific Pretraining of Language Models: A Comparative Study in the Medical Field

2024/07/19 by Kerner, Tobias · 1 citation
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.6 #I.2.7 #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2407.14076

Abstract

There are many cases where LLMs are used for specific tasks in a single domain. These usually require less general, but more domain-specific knowledge. Highly capable, general-purpose state-of-the-art language models like GPT-4 or Claude-3-opus can often be used for such tasks, but they are very large and cannot be run locally, even if they were not proprietary. This can be a problem when working with sensitive data. This paper focuses on domain-specific and mixed-domain pretraining as potentially more efficient methods than general pretraining for specialized language models. We will take a look at work related to domain-specific pretraining, specifically in the medical area, and compare benchmark results of specialized language models to general-purpose language models.

Cited by

• From Amateur to Master: Infusing Knowledge into LLMs via Automated Curriculum Learning

Related

• MDAPT: Multilingual Domain Adaptive Pretraining in a Single Model
• A Survey on Large Language Models from General Purpose to Medical Applications: Datasets, Methodologies, and Evaluations
• Do We Still Need Clinical Language Models?
• Small or Large? Zero-Shot or Finetuned? Guiding Language Model Choice for Specialized Applications in Healthcare
• Beyond Fine-tuning: Unleashing the Potential of Continuous Pretraining for Clinical LLMs