Boosting Large Language Models with Mask Fine-Tuning

2025/03/27 by Zhang, Mingyuan, Bai, Yue, Wang, Huan +3 · 1 citation
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2503.22764

Abstract

The model is usually kept integral in the mainstream large language model (LLM) fine-tuning protocols. No works have questioned whether maintaining the integrity of the model is indispensable for performance. In this work, we introduce Mask Fine-Tuning (MFT), a brand-new LLM fine-tuning paradigm to show that properly breaking the integrity of the model can surprisingly lead to improved performance. Specifically, MFT learns a set of binary masks supervised by the typical LLM fine-tuning objective. Extensive experiments show that MFT gains a consistent performance boost across various domains and backbones (e.g., 1.95%/1.88% average gain in coding with LLaMA2-7B/3.1-8B). Detailed procedures are provided to study the proposed MFT from different hyperparameter perspectives for better insight. In particular, MFT naturally updates the current LLM training protocol by deploying it on a complete well-trained model. This study extends the functionality of mask learning from its conventional network pruning context for model compression to a more general scope.

Cited by

• Rethinking Fine-Tuning: Unlocking Hidden Capabilities in Vision-Language Models

Related

• Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuning
• MaskPrune: Mask-based LLM Pruning for Layer-wise Uniform Structures
• Masking as an Efficient Alternative to Finetuning for Pretrained Language Models
• MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models
• Should You Mask 15% in Masked Language Modeling?