Scaling Laws for Precision

2024/11/07 by Tanishq Kumar, Kumar, Tanishq, Zachary Ankner +16 · 7 voices · 29 citations
Arts and Humanities · #Philosophy and History of Science

paper · pdf · doi:10.48550/arxiv.2411.04330

Abstract

Low precision training and inference affect both the quality and cost of language models, but current scaling laws do not account for this. In this work, we devise "precision-aware" scaling laws for both training and inference. We propose that training in lower precision reduces the model's "effective parameter count," allowing us to predict the additional loss incurred from training in low precision and post-train quantization. For inference, we find that the degradation introduced by post-training quantization increases as models are trained on more data, eventually making additional pretraining data actively harmful. For training, our scaling laws allow us to predict the loss of a model with different parts in different precisions, and suggest that training larger models in lower precision may be compute optimal. We unify the scaling laws for post and pretraining quantization to arrive at a single functional form that predicts degradation from training and inference in varied precisions. We fit on over 465 pretraining runs and validate our predictions on model sizes up to 1.7B parameters trained on up to 26B tokens.

Cited by

• Not All Bits Are Equal: Scale-Dependent Memory Optimization Strategies for Reasoning Models
• Selecting Language Models for Social Science: Start Small, Start Open, and Validate
• When Does Learning Renormalize? Sufficient Conditions for Power Law Spectral Dynamics
• Renormalizable Spectral-Shell Dynamics as the Origin of Neural Scaling Laws
• Data Curation Through the Lens of Spectral Dynamics: Static Limits, Dynamic Acceleration, and Practical Oracles
• A Generalized Spectral Framework to Expain Neural Scaling and Compression Dynamics
• INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats
• Improving the Straight-Through Estimator with Zeroth-Order Information
• PTPP-Aware Adaptation Scaling Laws: Predicting Domain-Adaptation Performance at Unseen Pre-Training Budgets
• CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training
• Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs
• Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory
• Automating Structural Engineering Workflows with Large Language Model Agents
• xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexity
• On the Quantization Robustness of Diffusion Language Models in Coding Benchmarks
• SQS: Bayesian DNN Compression through Sparse Quantized Sub-distributions
• Training Dynamics Impact Post-Training Quantization Robustness
• Hyperparameter Loss Surfaces Are Simple Near their Optima
• CAST: Continuous and Differentiable Semi-Structured Sparsity-Aware Training for Large Language Models
• Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMs
• Scaling with Collapse: Efficient and Predictable Training of LLM Families
• Model Merging Scaling Laws in Large Language Models
• Evaluating the Robustness of Chinchilla Compute-Optimal Scaling
• Scaling LLM Test-Time Compute with Mobile NPU on Smartphones
• Compute-Optimal Quantization-Aware Training
• Universal Properties of Activation Sparsity in Modern Large Language Models
• Pushing the Envelope of LLM Inference on AI-PC
• Generalizing Scaling Laws for Dense and Sparse Large Language Models
• KLLM: Fast LLM Inference with K-Means Quantization

Discussions

• Scaling Laws for Precision yes, llama models are harder to quantize. They’re “overtrained”, on more data, so quantization removes a lot of critical information arxiv.org/abs/2411.04330 [bsky, 22 points, 1 comments]
• This recent paper arxiv.org/abs/2411.04330 suggests that it’s model specific depending on their size and number of tokens trained on. A small model trained with lots of tokens doesn’t compress as well [bsky, 16 points, 1 comments]
• Scaling Laws for Precision arxiv.org/abs/2411.04330 From Tanishq Kumar, Aditi Ragunathan et al. Studies effects of precision & quantization of parameters for training and inference in DL / language [bsky, 8 points, 0 comments]
• Scaling Laws for Precision [hn, 2 points, 0 comments]
• AI- Lot's of imprecise data, quantization to lower the number of bits, in a large model can be worse than a small model with precise data. arxiv.org/pdf/2411.04330 [bsky, 0 points, 0 comments]
• Thx. So if post-training quant was abandoned (relevant: arxiv.org/pdf/2411.04330), there might be new rationale for low-power (not just low latency) phys compute? Aside: holding params constant, I wou [bsky, 0 points, 1 comments]
• Pick of the week @fbk-mt.bsky.social Super interesting paper on the limitations of quantization, demonstrating how post-training quantization scales poorly in data. arxiv.org/abs/2411.04330 [bsky, 0 points, 0 comments]

Related

• Language models scale reliably with over-training and on downstream tasks
• Scaling Parameter-Constrained Language Models with Quality Data
• Scaling Inference-Efficient Language Models
• A Hitchhiker's Guide to Scaling Law Estimation
• Scaling Laws Revisited: Modeling the Role of Data Quality in Language Model Pretraining