Progtuning: Progressive Fine-tuning Framework for Transformer-based Language Models

2025/06/26 by Xihan Ji, Ji, Xiaoshuang, Zhendong Zhao +6
Computer Science · #Adaptability #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Domain adaptation #FOS: Computer and information sciences #Language model #Language understanding #Natural Language Processing Techniques #Scheduling (production processes) #Topic Modeling #Transformer

paper · pdf · doi:10.48550/arxiv.2506.21119

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/06/26 · openalex created_date 2025/10/15 · openalex updated_date 2026/08/05

Abstract

Fine-tuning is a promising technique for leveraging Transformer-based language models in downstream tasks. As model sizes continue to grow, updating all model parameters becomes increasingly costly. Parameter-efficient fine-tuning methods effectively address this issue by selectively updating a small subset of parameters. However, fine-tuning and most existing parameter-efficient fine-tuning methods require updating the same number of parameters as the initial size, ignoring the unequal contribution across Transformer blocks and leading to extremely inefficient allocation of computing resources. In this paper, we propose Progtuning, the novel fine-tuning framework combined with progressive learning for Transformer-based language models. Specifically, Progtuning progressively reduces the number of updated transformer blocks based on the contribution. Remarkably, Progtuning optimizes resource allocation and reduces the number of updated parameters by approximately 25%, while still maintaining competitive performance. And it also exhibits high adaptability with parameter-efficient fine-tuning methods, demonstrating excellent performance across various adaptation scenarios.

Citations

• DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distribution
• Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey
• LLM-FP4: 4-Bit Floating-Point Quantized Transformers
• LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models
• A Simple and Effective Pruning Approach for Large Language Models
• Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning
• LLaMA: Open and Efficient Foundation Language Models
• SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
• KronA: Parameter Efficient Tuning with Kronecker Adapter
• P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
• BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
• LoRA: Low-Rank Adaptation of Large Language Models
• The Power of Scale for Parameter-Efficient Prompt Tuning
• Parameter-Efficient Transfer Learning with Diff Pruning
• Language Models are Few-Shot Learners
• Cascade EF-GAN: Progressive Facial Expression Editing with Local Focuses
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• Parameter-Efficient Transfer Learning for NLP
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• A Fully Progressive Approach to Single-Image Super-Resolution
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• Layer Normalization
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Deep Residual Learning for Image Recognition
• Semi-supervised Sequence Learning
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• SPAFIT: Stratified Progressive Adaptation Fine-tuning for Pre-trained Large Language Models
• Can pruning make Large Language Models more efficient?
• Solo Connection: A Parameter Efficient Fine-Tuning Technique for Transformers
• SPT: Fine-Tuning Transformer-based Language Models Efficiently with Sparsification
• Primer: Searching for Efficient Transformers for Language Modeling