LoRA ensembles for large language model fine-tuning

2023/09/29 by Xi Wang, Laurence Aitchison, Wang, Xi +3 · 9 citations
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2310.00035

openalex publication_date 2023/09/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Finetuned LLMs often exhibit poor uncertainty quantification, manifesting as overconfidence, poor calibration, and unreliable prediction results on test data or out-of-distribution samples. One approach commonly used in vision for alleviating this issue is a deep ensemble, which constructs an ensemble by training the same model multiple times using different random initializations. However, there is a huge challenge to ensembling LLMs: the most effective LLMs are very, very large. Keeping a single LLM in memory is already challenging enough: keeping an ensemble of e.g. 5 LLMs in memory is impossible in many settings. To address these issues, we propose an ensemble approach using Low-Rank Adapters (LoRA), a parameter-efficient fine-tuning technique. Critically, these low-rank adapters represent a very small number of parameters, orders of magnitude less than the underlying pre-trained model. Thus, it is possible to construct large ensembles of LoRA adapters with almost the same computational overhead as using the original model. We find that LoRA ensembles, applied on its own or on top of pre-existing regularization techniques, gives consistent improvements in predictive accuracy and uncertainty quantification.

Cited by

• How (Mis)calibrated is Your Federated CLIP and What To Do About It?
• As If We've Met Before: LLMs Exhibit Certainty in Recognizing Seen Files
• Neural Diversity Regularizes Hallucinations in Small Models
• Ensembling Pruned Attention Heads For Uncertainty-Aware Efficient Transformers
• From Detection to Mitigation: Addressing Gender Bias in Chinese Texts via Efficient Tuning and Voting-Based Rebalancing
• SAGE: Scale-Aware Gradual Evolution for Continual Knowledge Graph Embedding
• Resource-Aware Aggregation and Sparsification in Heterogeneous Ensemble Federated Learning
• Human-Alignment and Calibration of Inference-Time Uncertainty in Large Language Models
• Multiple Choice Learning of Low Rank Adapters for Language Modeling

Related

• Uncertainty quantification in fine-tuned LLMs using LoRA ensembles
• C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Models
• Efficient Ensemble for Fine-tuning Language Models on Multiple Datasets
• Minimal Ranks, Maximum Confidence: Parameter-efficient Uncertainty Quantification for LoRA
• Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models