MaP: A Unified Framework for Reliable Evaluation of Pre-training Dynamics

2025/10/10 by Jiapeng Wang, Changxin Tian, Wang, Jiapeng +13
Computer Science · Medicine · #Topic Modeling #Artificial Intelligence in Healthcare and Education #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2510.09295

Abstract

Reliable evaluation is fundamental to the progress of Large Language Models (LLMs), yet the evaluation process during pre-training is plagued by significant instability that obscures true learning dynamics. In this work, we systematically diagnose this instability, attributing it to two distinct sources: Parameter Instability from training stochasticity and Evaluation Instability from noisy measurement protocols. To counteract both sources of noise, we introduce MaP, a dual-pronged framework that synergistically integrates checkpoint \underlineMerging \underlineand the \underlinePass@k metric. Checkpoint merging smooths the parameter space by averaging recent model weights, while Pass@k provides a robust, low-variance statistical estimate of model capability. Extensive experiments show that MaP yields significantly smoother performance curves, reduces inter-run variance, and ensures more consistent model rankings. Ultimately, MaP provides a more reliable and faithful lens for observing LLM training dynamics, laying a crucial empirical foundation for LLM research.

Citations

• Instability in Downstream Task Performance During LLM Pretraining
• A Survey on Large Language Model Benchmarks
• WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training
• Model Merging in Pre-training of Large Language Models
• Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
• BOSE: A Systematic Evaluation Method Optimized for Base Models
• GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
• The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism
• WARP: On the Benefits of Weight Averaged Rewarded Policies
• Post-Hoc Reversal: Are We Selecting Models Prematurely?
• Evaluating Mathematical Reasoning Beyond Accuracy
• Checkpoint Merging via Bayesian Optimization in LLM Pretraining
• GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers
• CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?
• CMMLU: Measuring massive multitask language understanding in Chinese
• Early Weight Averaging meets High Learning Rates for LLM Pre-training
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
• C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models
• Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
• AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
• A Survey of Large Language Models
• Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averaging
• Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
• Training Verifiers to Solve Math Word Problems
• Evaluating Large Language Models Trained on Code
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Measuring Mathematical Problem Solving With the MATH Dataset
• Accounting for Variance in Machine Learning Benchmarks
• Underspecification Presents Challenges for Credibility in Modern Machine Learning
• Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel
• Measuring Massive Multitask Language Understanding
• Why gradient clipping accelerates training: A theoretical justification for adaptivity
• Know What You Don't Know: Unanswerable Questions for SQuAD
• Averaging Weights Leads to Wider Optima and Better Generalization
• Critical Learning Periods in Deep Neural Networks
• Three Factors Influencing Minima in SGD
• RACE: Large-scale ReAding Comprehension Dataset From Examinations
• A NEW MEASURE OF RANK CORRELATION

Related

• Stable Language Model Pre-training by Reducing Embedding Variability
• FTFT: Efficient and Robust Fine-Tuning by Transferring Training Dynamics
• Early Weight Averaging meets High Learning Rates for LLM Pre-training
• Instability in Downstream Task Performance During LLM Pretraining
• The Fine Line: Navigating Large Language Model Pretraining with Down-streaming Capability Analysis