Scaling Laws for Transfer

2021/02/02 by Danny Hernandez, Jared Kaplan, Hernandez, Danny +5 · 1 voice · 50 citations
Computer Science · Mathematics · #Artificial intelligence #Computer science #Domain Adaptation and Few-Shot Learning #Entropy (arrow of time) #FOS: Computer and information sciences #Generality #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Mathematics #Physics #Power law #Scaling #Scaling law #Scratch #Statistics #Topic Modeling #Transfer of learning #cs.LG

paper · pdf · doi:10.48550/arxiv.2102.01293

published in arXiv (Cornell University) (Cornell University) · 19 pages, 15 figures

arxiv created 2021/02/02 · openalex publication_date 2021/02/02 · arxiv published 2021/02/02 · arxiv updated 2021/02/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We study empirical scaling laws for transfer learning between distributions in an unsupervised, fine-tuning setting. When we train increasingly large neural networks from-scratch on a fixed-size dataset, they eventually become data-limited and stop improving in performance (cross-entropy loss). When we do the same for models pre-trained on a large language dataset, the slope in performance gains is merely reduced rather than going to zero. We calculate the effective data "transferred" from pre-training by determining how much data a transformer of the same size would have required to achieve the same loss when training from scratch. In other words, we focus on units of data while holding everything else fixed. We find that the effective data transferred is described well in the low data regime by a power-law of parameter count and fine-tuning dataset size. We believe the exponents in these power-laws correspond to measures of the generality of a model and proximity of distributions (in a directed rather than symmetric sense). We find that pre-training effectively multiplies the fine-tuning dataset size. Transfer, like overall performance, scales predictably in terms of parameters, data, and compute.

Citations

• Scaling Laws for Neural Language Models
• Learning to summarize from human feedback
• Scaling Laws for Autoregressive Generative Modeling
• Learning Transferable Visual Models From Natural Language Supervision

Cited by

• From Zipf's Law to Neural Scaling through Heaps' Law and Hilberg's Hypothesis
• Prompting-in-a-Series: Psychology-Informed Contents and Embeddings for Personality Recognition With Decoder-Only Models
• Efficient Inference Using Large Language Models with Limited Human Data: Fine-Tuning then Rectification
• Data Value in the Age of Scaling: Understanding LLM Scaling Dynamics Under Real-Synthetic Data Mixtures
• Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
• CoT-X: An Adaptive Framework for Cross-Model Chain-of-Thought Transfer and Optimization
• xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexity
• Reusing Overtrained Language Models Saturates Scaling
• Efficient Prediction of Pass@k Scaling in Large Language Models
• What Scales in Cross-Entropy Scaling Law?
• Emergent World Representations in OpenVLA
• Where Did It Go Wrong? Attributing Undesirable LLM Behaviors via Representation Gradient Tracing
• Training Larger Networks for Deep Reinforcement Learning
• Retrieval Capabilities of Large Language Models Scale with Pretraining FLOPs
• Large Language Model Scaling Laws for Neural Quantum States in Quantum Chemistry
• Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models
• Scaling and Data Saturation in Protein Language Models
• Can Language Models Discover Scaling Laws?
• Scaling Scaling Laws with Board Games
• A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Search
• Sub-Scaling Laws: On the Role of Data Density and Training Strategies in LLMs
• Scaling Laws for Optimal Data Mixtures
• Disentangling Transfer and Interference in Multi-Domain Learning
• Online Handbook of Argumentation for AI: Volume 2
• LRM-1B: Towards Large Routing Model
• Doc2SAR: A Synergistic Framework for High-Fidelity Extraction of Structure-Activity Relationships from Scientific Documents
• A Scaling Law for Synthetic-to-Real Transfer: How Much Is Your Pre-training Effective?
• From Data to Knowledge: Evaluating How Efficiently Language Models Learn Facts
• Pretraining Representations for Data-Efficient Reinforcement Learning
• Scaling Laws of Motion Forecasting and Planning -- Technical Report
• Complexity Scaling Laws for Neural Models using Combinatorial Optimization
• The Efficiency Misnomer
• Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Models
• Training Cross-Morphology Embodied AI Agents: From Practical Challenges to Theoretical Foundations
• Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
• GUST: Quantifying Free-Form Geometric Uncertainty of Metamaterials Using Small Data
• Large Language Models' Reasoning Stalls: An Investigation into the Capabilities of Frontier Models
• Inference Compute-Optimal Video Vision Language Models
• AdaSTaR: Adaptive Data Sampling for Training Self-Taught Reasoners
• CoLA: Collaborative Low-Rank Adaptation
• Exploring the Limits of Large Scale Pre-training
• Learning Dynamics in Continual Pre-Training for Large Language Models
• A Scaling Law for Token Efficiency in LLM Fine-Tuning Under Fixed Compute Budgets
• Position: Enough of Scaling LLMs! Lets Focus on Downscaling
• Time is Not Compute: Scaling Laws for Wall-Clock Constrained Training on Consumer GPUs
• LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection
• Finding the Minimal Parameter Budget for Implicit Reasoning: A Data Complexity Driven Scaling Law for Language Models
• Efficient Pretraining Length Scaling
• Scaling Laws for Data-Efficient Visual Transfer Learning
• On Model and Data Scaling for Skeleton-based Self-Supervised Gait Recognition

Discussions

• OpenAI – Scaling Laws for Transfer [hn, 3 points, 0 comments]

Related

• An Empirical Study of Scaling Laws for Transfer
• Scaling Laws for Neural Language Models
• Features are fate: a theory of transfer learning in high-dimensional regression
• Scaling Laws for Downstream Task Performance of Large Language Models
• A Dynamical Model of Neural Scaling Laws