Bridging Information-Theoretic and Geometric Compression in Language Models

2023/10/20 by Emily Cheng, Cheng, Emily, Corentin Kervadec +3 · 2 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2310.13620

openalex publication_date 2023/10/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

For a language model (LM) to faithfully model human language, it must compress vast, potentially infinite information into relatively few dimensions. We propose analyzing compression in (pre-trained) LMs from two points of view: geometric and information-theoretic. We demonstrate that the two views are highly correlated, such that the intrinsic geometric dimension of linguistic data predicts their coding length under the LM. We then show that, in turn, high compression of a linguistic dataset predicts rapid adaptation to that dataset, confirming that being able to compress linguistic information is an important part of successful LM performance. As a practical byproduct of our analysis, we evaluate a battery of intrinsic dimension estimators for the first time on linguistic data, showing that only some encapsulate the relationship between information-theoretic compression, geometric compression, and ease-of-adaptation.

Cited by

• Geometry of Decision Making in Language Models
• Controlling the Risk of Corrupted Contexts for Language Models via Early-Exiting

Related

• Compression Hacking: A Supplementary Perspective on Informatics Properties of Language Models from Geometric Distortion
• Compression Laws for Large Language Models
• Understanding LLM Behaviors via Compression: Data Generation, Knowledge Acquisition and Scaling Laws
• Ranking LLMs by compression
• The Information of Large Language Model Geometry