PLoP: Precise LoRA Placement for Efficient Finetuning of Large Models

2025/06/25 by Soufiane Hayou, Hayou, Soufiane, N. C. Ghosh +3 · 2 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Model-Driven Software Engineering Techniques #Natural Language Processing Techniques #Time Series Analysis and Forecasting

paper · pdf · doi:10.48550/arxiv.2506.20629

openalex publication_date 2025/06/25 · openalex created_date 2025/10/09 · openalex updated_date 2026/07/28

Abstract

Low-Rank Adaptation (LoRA) is a widely used finetuning method for large models. Its small memory footprint allows practitioners to adapt large models to specific tasks at a fraction of the cost of full finetuning. Different modifications have been proposed to enhance its efficiency by, for example, setting the learning rate, the rank, and the initialization. Another improvement axis is adapter placement strategy: when using LoRA, practitioners usually pick module types to adapt with LoRA, such as Query and Key modules. Few works have studied the problem of adapter placement, with nonconclusive results: original LoRA paper suggested placing adapters in attention modules, while other works suggested placing them in the MLP modules. Through an intuitive theoretical analysis, we introduce PLoP (Precise LoRA Placement), a lightweight method that allows automatic identification of module types where LoRA adapters should be placed, given a pretrained model and a finetuning task. We demonstrate that PLoP consistently outperforms, and in the worst case competes, with commonly used placement strategies through comprehensive experiments on supervised finetuning and reinforcement learning for reasoning.

Citations

• The Impact of Initialization on LoRA Finetuning Dynamics
• Understanding and Minimising Outlier Features in Neural Network Training
• Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning
• HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning
• A Note on LoRA
• LoRA+: Efficient Low Rank Adaptation of Large Models
• DoRA: Weight-Decomposed Low-Rank Adaptation
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models
• QLoRA: Efficient Finetuning of Quantized LLMs
• AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
• Sensitivity-Aware Visual Parameter-Efficient Fine-Tuning
• Training Verifiers to Solve Math Word Problems
• Towards a Unified View of Parameter-Efficient Transfer Learning
• Evaluating Large Language Models Trained on Code
• Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation
• Feature Learning in Infinite-Width Neural Networks
• Measuring Massive Multitask Language Understanding
• Adversarial NLI: A New Benchmark for Natural Language Understanding
• On the Impact of the Activation Function on Deep Neural Networks\n Training
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• signSGD: Compressed Optimisation for Non-Convex Problems
• TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for\n Reading Comprehension
• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
• Take Only What You Need: Rank Minimization as an Implicit Forgetting Regularizer in Continual Learning
• LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuning
• CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge

Cited by

• α-LoRA: Effective Fine-Tuning via Base Model Rescaling
• Training-Free versus Training-Based Intent Classification in LLMs: Accuracy, Robustness, and Failure Modes

Related

• PLoRA: Efficient LoRA Hyperparameter Tuning for Large Models
• A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA
• LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuning
• L1RA: Dynamic Rank Assignment in LoRA Fine-Tuning
• LoRA Learns Less and Forgets Less