Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models

2025/08/16 by Yuan Li, Zhengzhong Liu, Li, Yuan +3 · 1 citation
Computer Science · #Data point #Domain (mathematical analysis) #Frame (networking) #Grid #Mixing (physics) #Model selection #Natural Language Processing Techniques #Reduction (mathematics) #Selection (genetic algorithm) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2508.11953

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/08/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Optimizing data mixtures for supervised fine-tuning (SFT) of large language models (LLMs) is critical for developing general-purpose models, yet this area remains underexplored. In this paper, we frame data mixing as an optimization problem and introduce a novel method designed to minimize validation loss. Our approach parametrizes the loss by modeling effective data transferred and leveraging scaling laws for fine-tuning. By experimenting with various small-scale data mixtures, we fit these parameters and derive the optimal weights. We provide both mathematical proofs and empirical results demonstrating that our algorithm achieves excellent overall and individual performance across all domains. Through controlled experiments, we show that models trained with our optimized weights perform on par with those using optimal weights determined via grid search, with per-domain loss only 0.66% higher than the best domain loss from grid search on average. Additionally, we show that reweighting popular SFT datasets using our method improves both validation loss and downstream performance. Finally, we discuss how our method can generalize to guide data selection for domain-specific models and provide insights into SFT.

Citations

• Tulu 3: Pushing Frontiers in Open Language Model Post-Training
• Aioli: A Unified Optimization Framework for Language Model Data Mixing
• OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models
• FineTuneBench: How well do commercial fine-tuning APIs infuse knowledge into LLMs?
• Scaling Laws for Predicting Downstream Performance in LLMs
• OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data
• Mixing It Up: The Cocktail Effect of Multi-Task Fine-Tuning on LLM Performance -- A Case Study in Finance
• Revisiting the Superficial Alignment Hypothesis
• Beyond IID: Optimizing Instruction Learning from the Perspective of Instruction Interaction and Dependency
• RegMix: Data Mixture as Regression for Language Model Pre-training
• Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance
• SMART: Submodular Data Mixture Strategy for Instruction Tuning
• When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method
• Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning
• Efficient Online Data Mixing For Language Model Pre-Training
• Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2
• Instruction-Following Evaluation for Large Language Models
• How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition
• How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources
• Orca: Progressive Learning from Complex Explanation Traces of GPT-4
• AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
• Instruction Tuning with GPT-4
• LLaMA: Open and Efficient Foundation Language Models
• The Flan Collection: Designing Data and Methods for Effective Instruction Tuning
• Scaling Instruction-Finetuned Language Models
• Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
• Training Compute-Optimal Large Language Models
• How Many Data Samples is an Additional Instruction Worth?
• ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection
• PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts
• Training Verifiers to Solve Math Word Problems
• TruthfulQA: Measuring How Models Mimic Human Falsehoods
• Finetuned Language Models Are Zero-Shot Learners
• Evaluating Large Language Models Trained on Code
• Scaling Laws for Transfer
• Measuring Massive Multitask Language Understanding
• PubMedQA: A Dataset for Biomedical Research Question Answering
• HellaSwag: Can a Machine Really Finish Your Sentence?
• Выпуклая оптимизация
• On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming
• AutoScale: Scale-Aware Data Mixing for Pre-Training LLMs

Cited by

• Learning Task Mixtures from Task Affinities: A Probabilistic Graphical Model for Supervised Fine-Tuning
• Multi-task Code LLMs: Data Mix or Model Merge?

Related

• Rethinking Data Mixture for Large Language Models: A Comprehensive Survey and New Perspectives
• Mitigating Training Imbalance in LLM Fine-Tuning via Selective Parameter Merging
• Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance
• DomainPilot: Domain-Level Loss-Guided Two-Stage Data Mixture Optimization for Efficient Language Model Fine-Tuning
• Rethinking Data Selection at Scale: Random Selection is Almost All You Need