Tracing the Representation Geometry of Language Models from Pretraining to Post-training

2025/09/27 by Melody Zixuan Li, Minwei Li, Li, Melody Zixuan +13 · 2 voices · 4 citations
Computer Science · #Natural Language Processing Techniques #Topic Modeling #cs.AI #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2509.23024

openalex publication_date 2025/09/27 · openalex created_date 2025/10/19 · openalex updated_date 2026/07/28

Abstract

Standard training metrics like loss fail to explain the emergence of complex capabilities in large language models. We take a spectral approach to investigate the geometry of learned representations across pretraining and post-training, measuring effective rank (RankMe) and eigenspectrum decay (α-ReQ). With OLMo (1B-7B) and Pythia (160M-12B) models, we uncover a consistent non-monotonic sequence of three geometric phases during autoregressive pretraining. The initial "warmup" phase exhibits rapid representational collapse. This is followed by an "entropy-seeking" phase, where the manifold's dimensionality expands substantially, coinciding with peak n-gram memorization. Subsequently, a "compression-seeking" phase imposes anisotropic consolidation, selectively preserving variance along dominant eigendirections while contracting others, a transition marked with significant improvement in downstream task performance. We show these phases can emerge from a fundamental interplay of cross-entropy optimization under skewed token frequencies and representational bottlenecks (d ≪ |V|). Post-training further transforms geometry: SFT and DPO drive "entropy-seeking" dynamics to integrate specific instructional or preferential data, improving in-distribution performance while degrading out-of-distribution robustness. Conversely, RLVR induces "compression-seeking", enhancing reward alignment but reducing generation diversity.

Citations

• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
• Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
• How do language models learn facts? Dynamics, curricula and hallucinations
• Overtrained Language Models Are Harder to Fine-Tune
• 2 OLMo 2 Furious
• The Complexity Dynamics of Grokking
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training
• The Llama 3 Herd of Models
• Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data
• Learning Dynamics of LLM Finetuning
• The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
• Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers
• Hallmarks of Optimization Trajectories in Neural Networks: Directional Exploration and Redundancy
• Neural Networks Learn Statistics of Increasing Complexity
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• OLMo: Accelerating the Science of Language Models
• Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens
• Harnessing small projectors and multiple views for efficient vision pretraining
• LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architectures
• The Transient Nature of Emergent In-Context Learning in Transformers
• Understanding the Inner Workings of Language Models Through Representation Dissimilarity
• Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regression
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
• On the Stepwise Nature of Self-Supervised Learning
• The Quantization Model of Neural Scaling
• Emergent Abilities of Large Language Models
• Training Compute-Optimal Large Language Models
• Investigating Power laws in Deep Representation Learning
• Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets
• Evaluating Large Language Models Trained on Code
• Learning Transferable Visual Models From Natural Language Supervision
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• Gradient Starvation: A Learning Proclivity in Neural Networks
• A Simple Framework for Contrastive Learning of Visual Representations
• Scaling Laws for Neural Language Models
• SPoC: Search-based Pseudocode to Code
• Representation Learning with Contrastive Predictive Coding
• Crowdsourcing Multiple Choice Science Questions
• TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for\n Reading Comprehension
• Pointer Sentinel Mixture Models

Cited by

• Context Structure Reshapes the Representational Geometry of Language Models
• Stable but Miscalibrated: A Kantian View on Overconfidence from Filters to Large Language Models
• The Geometry of Reasoning: Flowing Logics in Representation Space
• Convergence and Divergence of Language Models under Different Random Seeds
• Stop Spinning Wheels: Mitigating LLM Overthinking via Mining Patterns for Early Reasoning Exit
• Understanding Post-Training Structural Changes in Large Language Models

Discussions

• Also, even if a fixed distribution, it's only non-trivial if we think the system has been learning long enough to have converged. When we track representations in an LLM during pretraining they're con [bsky, 7 points, 1 comments]
• A big shoutout to @koustuvsinha.com for insightful discussions that shaped this work, and @natolambert.bsky.social + the OLMo team! Paper 📝: arxiv.org/abs/2509.23024 👩‍💻 Code : Coming soon! 👨‍💻 [bsky, 6 points, 0 comments]

Related

• Representational Curvature Modulates Behavioral Uncertainty in Large Language Models
• Scaling Laws for Post Training Quantized Large Language Models
• Memorization-Compression Cycles Improve Generalization
• CE-U: Cross Entropy Unlearning
• Scaling Laws for Neural Language Models