2024/02/27 by Pengjie Ren, Ren, Pengjie, Chengshun Shi +13 · 1 citation
Engineering · #68T50 #Advanced Adaptive Filtering Techniques #Advanced Wireless Communication Techniques #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.7 #Optical Network Technologies
paper · pdf · doi:10.48550/arxiv.2402.17263
openalex publication_date 2024/02/27 · openalex created_date 2024/03/05 · openalex updated_date 2026/07/28
Parameter-efficient fine-tuning (PEFT) is a popular method for tailoring pre-trained large language models (LLMs), especially as the models' scale and the diversity of tasks increase. Low-rank adaptation (LoRA) is based on the idea that the adaptation process is intrinsically low-dimensional, i.e., significant model changes can be represented with relatively few parameters. However, decreasing the rank encounters challenges with generalization errors for specific tasks when compared to full-parameter fine-tuning. We present MELoRA, a mini-ensemble low-rank adapters that uses fewer trainable parameters while maintaining a higher rank, thereby offering improved performance potential. The core idea is to freeze original pretrained weights and train a group of mini LoRAs with only a small number of parameters. This can capture a significant degree of diversity among mini LoRAs, thus promoting better generalization ability. We conduct a theoretical analysis and empirical studies on various NLP tasks. Our experimental results show that, compared to LoRA, MELoRA achieves better performance with 8 times fewer trainable parameters on natural language understanding tasks and 36 times fewer trainable parameters on instruction following tasks, which demonstrates the effectiveness of MELoRA.