Mapping 1,000+ Language Models via the Log-Likelihood Vector

2025/02/22 by Momose Oyama, Hiroaki Yamagiwa, Oyama, Momose +5 · 3 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2502.16173

openalex publication_date 2025/02/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

To compare autoregressive language models at scale, we propose using log-likelihood vectors computed on a predefined text set as model features. This approach has a solid theoretical basis: when treated as model coordinates, their squared Euclidean distance approximates the Kullback-Leibler divergence of text-generation probabilities. Our method is highly scalable, with computational cost growing linearly in both the number of models and text samples, and is easy to implement as the required features are derived from cross-entropy loss. Applying this method to over 1,000 language models, we constructed a "model map," providing a new perspective on large-scale model analysis.

Cited by

• Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality
• Revealing Language Model Trajectories via Kullback-Leibler Divergence
• Likelihood Variance as Text Importance for Resampling Texts to Map Language Models

Related

• Likelihood Variance as Text Importance for Resampling Texts to Map Language Models
• A Comprehensive Survey of Accelerated Generation Techniques in Large Language Models
• Large Language Models as Markov Chains
• Model-diff: A Tool for Comparative Study of Language Models in the Input Space
• Revealing Language Model Trajectories via Kullback-Leibler Divergence