SparseGrad: A Selective Method for Efficient Fine-tuning of MLP Layers

2024/10/09 by Viktoriia Chekalina, Anna Rudenko, Chekalina, Viktoriia +8 · 1 citation
Biochemistry, Genetics and Molecular Biology · Computer Science · Engineering · #DNA and Biological Computing #Cellular Automata and Applications #Photonic and Optical Devices

paper · pdf · doi:10.48550/arxiv.2410.07383

Abstract

The performance of Transformer models has been enhanced by increasing the number of parameters and the length of the processed text. Consequently, fine-tuning the entire model becomes a memory-intensive process. High-performance methods for parameter-efficient fine-tuning (PEFT) typically work with Attention blocks and often overlook MLP blocks, which contain about half of the model parameters. We propose a new selective PEFT method, namely SparseGrad, that performs well on MLP blocks. We transfer layer gradients to a space where only about 1% of the layer's elements remain significant. By converting gradients into a sparse structure, we reduce the number of updated parameters. We apply SparseGrad to fine-tune BERT and RoBERTa for the NLU task and LLaMa-2 for the Question-Answering task. In these experiments, with identical memory requirements, our method outperforms LoRA and MeProp, robust popular state-of-the-art PEFT approaches.

Cited by

• Faster and Memory-Efficient Training of Sequential Recommendation Models for Large Catalogs

Related

• Sparse Matrix in Large Language Model Fine-tuning
• AdaGradSelect: An adaptive gradient-guided layer selection method for efficient fine-tuning of SLMs
• SPT: Fine-Tuning Transformer-based Language Models Efficiently with Sparsification
• Sparse Gradient Compression for Fine-Tuning Large Language Models
• Layer-wise Importance Matters: Less Memory for Better Performance in Parameter-efficient Fine-tuning of Large Language Models