Rethinking Supervised Fine-Tuning: Emphasizing Key Answer Tokens for Improved LLM Accuracy

2025/12/24 by Shi, Xiaofeng, Kou, Qian, Li, Yuduo +1
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2512.21017

Abstract

With the rapid advancement of Large Language Models (LLMs), the Chain-of-Thought (CoT) component has become significant for complex reasoning tasks. However, in conventional Supervised Fine-Tuning (SFT), the model could allocate disproportionately more attention to CoT sequences with excessive length. This reduces focus on the much shorter but essential Key portion-the final answer, whose correctness directly determines task success and evaluation quality. To address this limitation, we propose SFTKey, a two-stage training scheme. In the first stage, conventional SFT is applied to ensure proper output format, while in the second stage, only the Key portion is fine-tuned to improve accuracy. Extensive experiments across multiple benchmarks and model families demonstrate that SFTKey achieves an average accuracy improvement exceeding 5% over conventional SFT, while preserving the ability to generate correct formats. Overall, this study advances LLM fine-tuning by explicitly balancing CoT learning with additional optimization on answer-relevant tokens.

Citations

• Forgetting: A New Mechanism Towards Better Large Language Model Fine-tuning
• SFT-GO: Supervised Fine-Tuning with Group Optimization for Large Language Models
• Long-Short Chain-of-Thought Mixture Supervised Fine-Tuning Eliciting Efficient Reasoning in Large Language Models
• Supervised Optimism Correction: Be Confident When LLMs Are Sure
• Token Weighting for Long-Range Language Modeling
• TokenSkip: Controllable Chain-of-Thought Compression in LLMs
• Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining
• Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
• From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning
• The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Training Verifiers to Solve Math Word Problems
• A Few More Examples May Be Worth Billions of Parameters
• Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering

Related

• Long-Short Chain-of-Thought Mixture Supervised Fine-Tuning Eliciting Efficient Reasoning in Large Language Models
• ReFT: Reasoning with Reinforced Fine-Tuning
• On the Impact of Fine-Tuning on Chain-of-Thought Reasoning
• Enhancing the Reasoning Capabilities of Small Language Models via\n Solution Guidance Fine-Tuning
• Thinking Preference Optimization