Adaptive parameter-efficient fine-tuning via Hessian-informed subset selection

2025/05/18 by Shiyun Xu, Xu, Shiyun, Zhiqi Bu +1 · 1 citation
Computer Science · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2505.12579

openalex publication_date 2025/05/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Parameter-efficient fine-tuning (PEFT) is a highly effective approach for adapting large pre-trained models to downstream tasks with minimal computational overhead. At the core, PEFT methods freeze most parameters and only trains a small subset (say <0.1% of total parameters). Notably, different PEFT methods select different subsets, resulting in varying levels of performance. This variation prompts a key question: how to effectively select the most influential subset to train? We formulate the subset selection as a multi-task problem: maximizing the performance and minimizing the number of trainable parameters. We leverage a series of transformations -- including ε-constraint method and second-order Taylor approximation -- to arrive at the classical 0-1 knapsack problem, which we solve through the lens of Pareto optimality. Consequently, we propose AdaPEFT, a Hessian-informed PEFT that adapts to various tasks and models, in which the selected subset empirically transfers across training horizons and model sizes.

Citations

• PEFT A2Z: Parameter-Efficient Fine-Tuning Survey for Large Language and Vision Models
• LoRA-GA: Low-Rank Adaptation with Gradient Approximation
• Gradient descent with generalized Newton's method
• LoRA Learns Less and Forgets Less
• Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey
• Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning
• LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models
• LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuning
• QLABGrad: a Hyperparameter-Free and Convergence-Guaranteed Scheme for Deep Learning
• P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
• BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
• The Power of Scale for Parameter-Efficient Prompt Tuning
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• Parameter-Efficient Transfer Learning for NLP
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Prefix-Tuning: Optimizing Continuous Prompts for Generation

Cited by

• GEM: A Scale-Aware and Distribution-Sensitive Sparse Fine-Tuning Framework for Effective Downstream Adaptation

Related

• FPS: Feedforward-based Parameter Selection For Efficient Fine-Tuning
• See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decomposition
• Look Within or Look Beyond? A Theoretical Comparison Between Parameter-Efficient and Full Fine-Tuning
• Dynamic Subset Tuning: Expanding the Operational Range of Parameter-Efficient Training for Large Language Models
• Easy Adaptation: An Efficient Task-Specific Knowledge Injection Method for Large Models in Resource-Constrained Environments