Revealing Language Model Trajectories via Kullback-Leibler Divergence

2025/05/21 by Kishino, Ryo, Takase, Yusuke, Oyama, Momose +2
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2505.15353

Abstract

A recently proposed method enables efficient estimation of the KL divergence between language models, including models with different architectures, by assigning coordinates based on log-likelihood vectors. To better understand the behavior of this metric, we systematically evaluate KL divergence across a wide range of conditions using publicly available language models. Our analysis covers comparisons between pretraining checkpoints, fine-tuned and base models, and layers via the logit lens. We find that trajectories of language models, as measured by KL divergence, exhibit a spiral structure during pretraining and thread-like progressions across layers. Furthermore, we show that, in terms of diffusion exponents, model trajectories in the log-likelihood space are more constrained than those in weight space.

Citations

• Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality
• PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs
• Mapping 1,000+ Language Models via the Log-Likelihood Vector
• Advanced Natural-based interaction for the ITAlian language: LLaMAntino-3-ANITA
• Release of Pre-Trained Models for the Japanese Language
• Configurable Safety Tuning of Language Models with Synthetic Preference Data
• sDPO: Don't Use Your Data All at Once
• ORPO: Monolithic Preference Optimization without Reference Model
• LAB: Large-Scale Alignment for ChatBots
• Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive
• Generative Representational Instruction Tuning
• Gemini Goes to Med School: Exploring the Capabilities of Multimodal Large Language Models on Medical Challenge Problems & Hallucinations
• Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation
• SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling
• LLaMAntino: LLaMA 2 Models for Effective Text Generation in Italian Language
• Orca 2: Teaching Small Language Models How to Reason
• Zephyr: Direct Distillation of LM Alignment
• UltraFeedback: Boosting Language Models with Scaled AI Feedback
• OpenChat: Advancing Open-source Language Models with Mixed-Quality Data
• Textbooks Are All You Need II: phi-1.5 technical report
• Lost in the Middle: How Language Models Use Long Contexts
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• QLoRA: Efficient Finetuning of Quantized LLMs
• Enhancing Chat Language Models by Scaling High-quality Instructional Conversations
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
• AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
• A Note on Normalized Emergence Timing (in Pythia Language Model Evaluations)
• Capabilities of GPT-4 on Medical Challenge Problems
• Eliciting Latent Predictions from Transformers with the Tuned Lens
• LLaMA: Open and Efficient Foundation Language Models
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
• Efficient Training of Language Models to Fill in the Middle
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• OPT: Open Pre-trained Transformer Language Models
• Training Verifiers to Solve Math Word Problems
• BBQ: A Hand-Built Bias Benchmark for Question Answering
• TruthfulQA: Measuring How Models Mimic Human Falsehoods
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
• Program Synthesis with Large Language Models
• Evaluating Large Language Models Trained on Code
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• Query-Key Normalization for Transformers
• RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language\n Models
• Measuring Massive Multitask Language Understanding
• Language Models are Few-Shot Learners
• Longformer: The Long-Document Transformer
• PIQA: Reasoning about Physical Commonsense in Natural Language
• Fast Transformer Decoding: One Write-Head is All You Need
• Quantifying the Carbon Emissions of Machine Learning
• HellaSwag: Can a Machine Really Finish Your Sentence?
• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
• Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
• Gender Bias in Coreference Resolution
• Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods
• Neural networks and rational functions
• TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for\n Reading Comprehension
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp\n Minima
• Benefits of depth in neural networks
• Estimators of Fractal Dimension: Assessing the Roughness of Time Series and Spatial Data
• Fractional Brownian Motions, Fractional Noises and Applications
• CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge

Related

• Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models
• Convergence and Divergence of Language Models under Different Random Seeds
• Better Estimation of the Kullback--Leibler Divergence Between Language Models
• Mapping 1,000+ Language Models via the Log-Likelihood Vector
• Sinkhorn Distance Minimization for Knowledge Distillation