Shadow-FT: Tuning Instruct Model via Training on Paired Base Model

2025/05/19 by Wu, Taiqiang, Yang, Runming, Li, Jiayi +4 · 4 citations
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2505.12716

Abstract

Large language models (LLMs) consistently benefit from further fine-tuning on various tasks. However, we observe that directly tuning the Instruct (i.e., instruction-tuned) models often leads to marginal improvements and even performance degeneration. Notably, paired Base models, the foundation for these Instruct variants, contain highly similar weight values (i.e., less than 2% on average for Llama 3.1 8B). The Base model tends to be a good learner yet a weak backbone without post-training. Therefore, we propose a novel Shadow-FT framework to tune the Instruct models by leveraging the corresponding Base models. The key insight is to fine-tune the Base model, and then directly graft the learned weight updates to the Instruct model. Our proposed Shadow-FT introduces no additional parameters, is easy to implement, and significantly improves performance. We conduct extensive experiments on tuning mainstream LLMs, such as Qwen 3 and Llama 3 series, and evaluate them across 19 benchmarks covering coding, reasoning, and mathematical tasks. Experimental results demonstrate that Shadow-FT consistently outperforms conventional full-parameter and parameter-efficient tuning approaches. Further analyses indicate that Shadow-FT can be applied to multimodal large language models (MLLMs) and combined with direct preference optimization~(DPO). Codes and weights are available at \hrefhttps://github.com/wutaiqiang/Shadow-FTGithub.

Citations

• The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning
• Z1: Efficient Test-time Scaling with Code
• Gemma 3 Technical Report
• Efficient Model Development through Fine-tuning Transfer
• LIMO: Less is More for Reasoning
• s1: Simple test-time scaling
• Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitate
• Kimi k1.5: Scaling Reinforcement Learning with LLMs
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs
• Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systems
• LLM-NEO: Parameter Efficient Knowledge Distillation for Large Language Models
• O1 Replication Journey: A Strategic Progress Report -- Part 1
• A Survey on the Honesty of Large Language Models
• LoCa: Logit Calibration for Knowledge Distillation
• ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understanding
• The Llama 3 Herd of Models
• Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies
• Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs
• Mixture-of-Subspaces in Low-Rank Adaptation
• Language Models Resist Alignment: Evidence From Data Compression
• MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
• RE-Adapt: Reverse Engineered Adaptation of Large Language Models
• LoRA Learns Less and Forgets Less
• Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models
• Large Language Models for Education: A Survey and Outlook
• LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models
• LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
• Yi: Open Foundation Models by 01.AI
• Tuning Language Models by Proxy
• DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
• The Falcon Series of Open Language Models
• MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark
• Chat Vector: A Simple Approach to Equip LLMs with Instruction Following and Model Alignment in New Languages
• Qwen Technical Report
• Instruction Tuning for Large Language Models: A Survey
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• TheoremQA: A Theorem-driven Question Answering dataset
• Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
• HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge
• LLaMA: Open and Efficient Foundation Language Models
• Editing Models with Task Arithmetic
• ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
• Training language models to follow instructions with human feedback
• Training Verifiers to Solve Math Word Problems
• P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
• Program Synthesis with Large Language Models
• Evaluating Large Language Models Trained on Code
• BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
• Are NLP Models really able to Solve Simple Math Word Problems?
• Measuring Mathematical Problem Solving With the MATH Dataset
• Measuring Massive Multitask Language Understanding
• Language Models are Few-Shot Learners
• Don't Stop Pretraining: Adapt Language Models to Domains and Tasks
• PubMedQA: A Dataset for Biomedical Research Question Answering
• DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Distilling the Knowledge in a Neural Network

Cited by

• Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Merging
• Revisiting Model Interpolation for Efficient Reasoning
• Timber: Training-free Instruct Model Refining with Base via Effective Rank
• UniPar: A Unified LLM-Based Framework for Parallel and Accelerated Code Translation in HPC

Related

• Exploring the Benefits of Training Expert Language Models over Instruction Tuning
• HFT: Half Fine-Tuning for Large Language Models
• Tuna: Instruction Tuning using Feedback from Large Language Models
• Small Language Models Improve Giants by Rewriting Their Outputs
• An Emulator for Fine-Tuning Large Language Models using Small Language Models