Reinforcement learning fine-tuning of language model for instruction following and math reasoning

2025/06/11 by Yifu Han, Han, Yifu, Zhang, Geo
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Reinforcement Learning in Robotics #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2506.21560

openalex publication_date 2025/06/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This study investigates the effectiveness of reinforcement learning (RL) fine-tuning techniques on a compact language model (Qwen2.5-0.5B Base) for two challenging tasks: instruction following and mathematical reasoning. We compare supervised fine-tuning (SFT), Direct Preference Optimization (DPO) using preference-labeled data, and Reinforce Leave-One-Out (RLOO) with reward models. Our experiments show that RLOO with DeBERTa reward modeling achieves the best alignment, while DPO provides strong and consistent results. For math reasoing tasks, synthetic data augmentation and best-of-N sampling with an external verifier significantly improve accuracy, showing the potential of combining fine-tuning with inference-time tools. This study highlights key trade-offs and practical strategies for training lightweight, task-aligned small-scale language models.

Citations

• Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
• RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
• Stream of Search (SoS): Learning to Search in Language
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Self-Refine: Iterative Refinement with Self-Feedback
• Toolformer: Language Models Can Teach Themselves to Use Tools
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• Step-wise Adaptive Integration of Supervised Fine-tuning and Reinforcement Learning for Task-Specific LLMs
• Boosting Accuracy and Efficiency of Budget Forcing in LLMs via Reinforcement Learning for Mathematical Reasoning
• Efficient Reinforcement Finetuning via Adaptive Curriculum Learning
• Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Models
• Effective Reinforcement Learning for Reasoning in Language Models