Similarity-Based Domain Adaptation with LLMs

2025/03/07 by He, Jie, Zhou, Wendi, Li, Xiang Lorraine +1
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2503.05281

Abstract

Unsupervised domain adaptation leverages abundant labeled data from various source domains to generalize onto unlabeled target data. Prior research has primarily focused on learning domain-invariant features across the source and target domains. However, these methods often require training a model using source domain data, which is time-consuming and can limit model usage for applications with different source data. This paper introduces a simple framework that utilizes the impressive generalization capabilities of Large Language Models (LLMs) for target data annotation without the need of source model training, followed by a novel similarity-based knowledge distillation loss. Our extensive experiments on cross-domain text classification reveal that our framework achieves impressive performance, specifically, 2.44% accuracy improvement when compared to the SOTA method.

Related

• Adapt in Contexts: Retrieval-Augmented Domain Adaptation via In-Context Learning
• LFM: Leveraging Foundation Models for Source-Free Universal Domain Adaptation
• Can Smaller LLMs do better? Unlocking Cross-Domain Potential through Parameter-Efficient Fine-Tuning for Text Summarization
• Universal Source-Free Domain Adaptation
• UDALM: Unsupervised Domain Adaptation through Language Modeling