The Trade-offs of Domain Adaptation for Neural Language Models

2021/09/21 by David Grangier, Dan Iter, Grangier, David +1 · 2 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.2109.10274

Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2022

openalex publication_date 2021/09/21 · arxiv created 2022/03/21 · arxiv updated 2022/03/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This work connects language model adaptation with concepts of machine learning theory. We consider a training setup with a large out-of-domain set and a small in-domain set. We derive how the benefit of training a model on either set depends on the size of the sets and the distance between their underlying distributions. We analyze how out-of-domain pre-training before in-domain fine-tuning achieves better generalization than either solution independently. Finally, we present how adaptation techniques based on data selection, such as importance sampling, intelligent data selection and influence functions, can be presented in a common framework which highlights their similarity and also their subtle differences.

Cited by

• Towards General Continuous Memory for Vision-Language Models
• Transferable text data distillation by trajectory matching

Related

• On the Complementarity of Data Selection and Fine Tuning for Domain Adaptation
• On the Domain Adaptation and Generalization of Pretrained Language Models: A Survey
• Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey
• A Brief Review of Domain Adaptation
• Distill, Adapt, Distill: Training Small, In-Domain Models for Neural Machine Translation