From Data to Knowledge: Evaluating How Efficiently Language Models Learn Facts

2025/06/20 by Christoph, Daniel, Ploner, Max, Haller, Patrick +1
Computer Science · Social Sciences · #Computation and Language (cs.CL) #Computational and Text Analysis Methods #FOS: Computer and information sciences #Machine Learning (cs.LG) #Text Readability and Simplification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2506.16912

openalex publication_date 2025/06/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Sample efficiency is a crucial property of language models with practical implications for training efficiency. In real-world text, information follows a long-tailed distribution. Yet, we expect models to learn and recall frequent and infrequent facts. Sample-efficient models are better equipped to handle this challenge of learning and retaining rare information without requiring excessive exposure. This study analyzes multiple models of varying architectures and sizes, all trained on the same pre-training data. By annotating relational facts with their frequencies in the training corpus, we examine how model performance varies with fact frequency. Our findings show that most models perform similarly on high-frequency facts but differ notably on low-frequency facts. This analysis provides new insights into the relationship between model architecture, size, and factual learning efficiency.

Citations

• A Hitchhiker's Guide to Scaling Law Estimation
• LM-PUB-QUIZ: A Comprehensive Framework for Zero-Shot Evaluation of Relational Knowledge in Language Models
• A Systematic Review on Long-Tailed Learning
• Large Language Models Make Sample-Efficient Recommender Systems
• Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
• Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competition
• BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Models
• On the Scaling Laws of Geographical Representation in Language Models
• LLaMA: Open and Efficient Foundation Language Models
• Large Language Models Struggle to Learn Long-Tail Knowledge
• Transformers are Sample-Efficient World Models
• Scaling Laws for Transfer
• How Much Knowledge Can You Pack Into the Parameters of a Language Model?
• Scaling Laws for Neural Language Models
• Language Models as Knowledge Bases?

Related

• Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models
• Understanding Fact Recall in Language Models: Why Two-Stage Training Encourages Memorization but Mixed Training Teaches Knowledge
• Factual Probing Is [MASK]: Learning vs. Learning to Recall
• How Do Large Language Models Acquire Factual Knowledge During Pretraining?
• Scaling Laws for Fact Memorization of Large Language Models