On the Utility of Domain-Adjacent Fine-Tuned Model Ensembles for Few-shot Problems

2024/06/19 by Md Ibrahim Ibne Alam, Parikshit Ram, Alam, Md Ibrahim Ibne +7
Engineering · Physics and Astronomy · #3D Shape Modeling and Analysis #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Model Reduction and Neural Networks #Tunneling and Rock Mechanics

paper · pdf · doi:10.48550/arxiv.2406.13720

openalex publication_date 2024/06/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) have been observed to perform well on a wide range of downstream tasks when fine-tuned on domain-specific data. However, such data may not be readily available in many applications, motivating zero-shot or few-shot approaches using domain-adjacent models. While several fine-tuned models for various tasks are available, finding an appropriate domain-adjacent model for a given task is often not straight forward. In this paper, we study DAFT-E, a framework that utilizes an Ensemble of Domain-Adjacent Fine-Tuned Foundation Models for few-shot problems. We show that for zero-shot problems, this ensembling method provides an accuracy performance close to that of the single best model. With few-shot problems, this performance improves further, at which point DEFT-E can outperform any single domain-adjacent model while requiring much less data for domain-specific fine-tuning.

Related

• Making Pre-trained Language Models Better Few-shot Learners
• Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
• Synergy over Discrepancy: A Partition-Based Approach to Multi-Domain LLM Fine-Tuning
• Scaled Prompt-Tuning for Few-Shot Natural Language Generation
• Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMs