Language Modeling Is Compression

2023/09/19 by Grégoire Delétang, Anian Ruoss, Delétang, Grégoire +22 · 12 voices · 49 citations
Computer Science · #Natural Language Processing Techniques #Topic Modeling #Speech Recognition and Synthesis

paper · pdf · doi:10.48550/arxiv.2309.10668

Abstract

It has long been established that predictive models can be transformed into lossless compressors and vice versa. Incidentally, in recent years, the machine learning community has focused on training increasingly large and powerful self-supervised (language) models. Since these large language models exhibit impressive predictive capabilities, they are well-positioned to be strong compressors. In this work, we advocate for viewing the prediction problem through the lens of compression and evaluate the compression capabilities of large (foundation) models. We show that large language models are powerful general-purpose predictors and that the compression viewpoint provides novel insights into scaling laws, tokenization, and in-context learning. For example, Chinchilla 70B, while trained primarily on text, compresses ImageNet patches to 43.4% and LibriSpeech samples to 16.4% of their raw size, beating domain-specific compressors like PNG (58.5%) or FLAC (30.3%), respectively. Finally, we show that the prediction-compression equivalence allows us to use any compressor (like gzip) to build a conditional generative model.

Cited by

• Intelligence from Learnable Novelty
• What does a Bayes-filtered transformer believe? A predictive Monte Carlo approach
• Large Language Model-Enhanced Multi-hop Parallel Image Semantic Communication
• Information Theory and Statistical Learning
• Position: Natural Language Should Not Fully Replace Formal Languages
• Is MoE Routing a Huffman Code? Discovering the Frequency-Diversity Law in Chain-of-Thought
• Sequential KV Cache Compression via Probabilistic Language Tries: Beyond the Per-Vector Shannon Limit
• The unreasonable effectiveness of pattern matching
• Cosine capital: Large language models and the embedding of all things
• Latent learning: episodic memory complements parametric learning by enabling flexible reuse of experiences
• Large Language Diffusion Models
• Efficient Reasoning with Hidden Thinking
• LLM-based Source Code Compression via Thresholded Symbol Ranking
• CrystalFormer-CSP: Thinking Fast and Slow for Crystal Structure Prediction
• VLIC: Vision-Language Models As Perceptual Judges for Human-Aligned Image Compression
• Spherical Leech Quantization for Visual Tokenization and Generation
• Shakespeare, Entropy and Educated Monkeys
• LeMat-GenBench: A Unified Evaluation Framework for Crystal Generative Models
• STCTS: Generative Semantic Compression for Ultra-Low Bitrate Speech via Explicit Text-Prosody-Timbre Decomposition
• Closed-Loop Transformers: Autoregressive Modeling as Iterative Latent Equilibrium
• WavefrontDiffusion: Dynamic Decoding Schedule for Improved Reasoning
• Know Your Limits: Entropy Estimation Modeling for Compression and Generalization
• Information Capacity: Evaluating the Efficiency of Large Language Models via Text Compression
• Training Language Models via Neural Cellular Automata
• The cost of language: tokenization as a metric of labor
• Spatially-Aligned Chroma from Luma Prediction for Lossless JPEG XS Raw Image Compression
• The Lossy Horizon: Error-Bounded Predictive Coding for Lossy Text Compression (Episode I)
• Explaining and Mitigating Crosslingual Tokenizer Inequities
• Human languages trade off complexity against efficiency
• Vector Quantization in the Brain: Grid-like Codes in World Models
• AI Agents as Universal Task Solvers
• The Algorithmic Regulator
• Benchmarking Language Modeling for Lossless Compression of Full-Fidelity Audio
• Latent Thinking Optimization: Your Latent Reasoning Language Model Secretly Encodes Reward Signals in Its Latent Thoughts
• Emergent Slow Thinking in LLMs as Inverse Tree Freezing
• Bridging Kolmogorov Complexity and Deep Learning: Asymptotically Optimal Description Length Objectives for Transformers
• A circuit for predicting hierarchical structure in-context in Large Language Models
• SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMs
• Lossless Compression: A New Benchmark for Time Series Model Evaluation
• SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass
• Predictable Compression Failures: Why Language Models Actually Hallucinate
• Way to Build Native AI-driven 6G Air Interface: Principles, Roadmap, and Outlook
• GLSim: Detecting Object Hallucinations in LVLMs via Global-Local Similarity
• Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training
• Revisiting MLLM Token Technology through the Lens of Classical Visual Coding
• How Causal Abstraction Underpins Computational Explanation
• PRELUDE: A Benchmark Designed to Require Global Comprehension and Reasoning over Long Contexts
• ByteGen: A Tokenizer-Free Generative Model for Orderbook Events in Byte Space
• Transmission With Machine Language Tokens: A Paradigm for Task-Oriented Agent Communication

Discussions

• Sorry! Wrong URL. I have seen this done a few different places and ways and the one I was actually thinking of was the gzip experiments of arxiv.org/abs/2309.10668 [bsky, 16 points, 1 comments]
• Related previous work. arxiv.org/abs/2309.10668 [bsky, 12 points, 0 comments]
• Language Modeling Is Compression [hn, 5 points, 0 comments]
• Language Modeling Is Compression [hn, 3 points, 1 comments]
• they compare gzip, png and other compression algos directly to LLMs and other transformers in this paper if you've never seen it: arxiv.org/abs/2309.10668 fun read [bsky, 3 points, 0 comments]
• (none of this is to say the supremely clever zip file isn't *useful*. But it is a *problem* that the SotA seems to require keeping a huge part of the training corpus, and *imo* indicative that the sto [bsky, 1 points, 0 comments]
• I think you are slowly coming to the insight. Now try to lose your narrow explanation of compression. And again, here is the google paper about this; arxiv.org/abs/2309.10668 [bsky, 0 points, 1 comments]
• This article explores AI advancements and implications, highlighting potential benefits and challenges. It emphasizes the importance of ethics and responsible AI development as the technology evolves. [bsky, 0 points, 0 comments]
• I've long believed that intelligence is compression. This paper supports that thesis: arxiv.org/pdf/2309.106.... [bsky, 0 points, 0 comments]
• The related work section of this paper from last year is quite useful. Prediction vs compression, effects of tokenization, etc… arxiv.org/pdf/2309.10668 [bsky, 0 points, 1 comments]
• arxiv.org/abs/2309.10668 [bsky, 0 points, 0 comments]
• "Consequently, we showed that the optimal model size is inextricably linked to the dataset size and cannot be scaled without limit." arxiv.org/pdf/2309.106... [bsky, 0 points, 0 comments]

Related

• Lossless Compression of Large Language Model-Generated Text via Next-Token Prediction
• Compression Laws for Large Language Models
• Large Language Models for Lossless Image Compression: Next-Pixel Prediction in Language Space is All You Need
• Evaluating Large Language Models for Generalization and Robustness via Data Compression
• Shifting AI Efficiency From Model-Centric to Data-Centric Compression