PhyloLM : Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarks

2024/04/06 by Nicolas Yax, Yax, Nicolas, Pierre-Yves Oudeyer +4 · 9 voices · 5 citations
Computer Science · #Biology #Computer science #Genetics #Natural Language Processing Techniques #Phylogenetics #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2404.04671

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/04/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

This paper introduces PhyloLM, a method adapting phylogenetic algorithms to Large Language Models (LLMs) to explore whether and how they relate to each other and to predict their performance characteristics. Our method calculates a phylogenetic distance metric based on the similarity of LLMs' output. The resulting metric is then used to construct dendrograms, which satisfactorily capture known relationships across a set of 111 open-source and 45 closed models. Furthermore, our phylogenetic distance predicts performance in standard benchmarks, thus demonstrating its functional validity and paving the way for a time and cost-effective estimation of LLM capabilities. To sum up, by translating population genetic concepts to machine learning, we propose and validate a tool to evaluate LLM development, relationships and capabilities, even in the absence of transparent training information.

Cited by

• LLM DNA: Tracing Model Evolution via Functional Representations
• Toward a Theory of Generalizability in LLM Mechanistic Interpretability Research
• Copyright Protection for Large Language Models: A Survey of Methods, Challenges, and Trends
• Likelihood Variance as Text Importance for Resampling Texts to Map Language Models
• Discovering Hidden Gems in Model Repositories

Discussions

• Inferring the Phylogeny of Large Language Models [hn, 69 points, 6 comments]
• In short, PhyloLM is a cheap and versatile algorithm that generates useful representations for LLMs that can have creative applications in pratice. 9/10 paper : arxiv.org/abs/2404.04671 colab : colab. [bsky, 2 points, 1 comments]
• Inferring the Phylogeny of Large Language Models #HackerNews https://arxiv.org/abs/2404.04671 [bsky, 1 points, 0 comments]
• Inferring the Phylogeny of Large Language Models https://arxiv.org/abs/2404.04671 (https://news.ycombinator.com/item?id=43736366) [bsky, 0 points, 0 comments]
• PhyloLM : Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarks http://arxiv.org/abs/2404.04671v2 [bsky, 0 points, 0 comments]
• ⚡ Hackernews Top story: Inferring the Phylogeny of Large Language Models [bsky, 0 points, 0 comments]
• Inferring the Phylogeny of Large Language Models https://arxiv.org/abs/2404.04671 [bsky, 0 points, 0 comments]
• Inferring the Phylogeny of Large Language Models https://arxiv.org/abs/2404.04671 (https://news.ycombinator.com/item?id=43736366) [bsky, 0 points, 0 comments]
• Inferring the Phylogeny of Large Language Models [bsky, 0 points, 0 comments]

Related

• PhyloGen: Language Model-Enhanced Phylogenetic Inference via Graph Structure Generation
• On the Origin of LLMs: An Evolutionary Tree and Graph for 15,821 Large Language Models
• LLM DNA: Tracing Model Evolution via Functional Representations
• Phyloformer: Fast, Accurate, and Versatile Phylogenetic Reconstruction with Deep Neural Networks
• The Cognate Data Bottleneck in Language Phylogenetics