Learning Curve Theory

2021/02/08 by Marcus Hutter · 2 voices · 5 citations
Computer Science · Mathematics · #cs.LG #stat.ML

paper · pdf

published as Latest 2021 version at http://www.hutter1.net/publ/scaling.pdf · 26 pages, 6 Figures

arxiv created 2021/02/08 · arxiv published 2021/02/08 · arxiv updated 2021/02/15

Abstract

Recently a number of empirical "universal" scaling law papers have been published, most notably by OpenAI. `Scaling laws' refers to power-law decreases of training or test error w.r.t. more data, larger neural networks, and/or more compute. In this work we focus on scaling w.r.t. data size n. Theoretical understanding of this phenomenon is largely lacking, except in finite-dimensional models for which error typically decreases with n^-1/2 or n^-1, where n is the sample size. We develop and theoretically analyse the simplest possible (toy) model that can exhibit n^-β learning curves for arbitrary power β>0, and determine whether power laws are universal or depend on the data distribution.

Cited by

• Quiet Feature Learning in Algorithmic Tasks
• Superposition Yields Robust Neural Scaling
• Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architectures
• Learning curves theory for hierarchically compositional data with power-law distributed features
• Deriving Neural Scaling Laws from the statistics of natural language

Discussions

• Learning Curve Theory [hn, 2 points, 0 comments]
• Learning Curve Theory [hn, 2 points, 0 comments]

Related

• Neural Scaling Laws Rooted in the Data Distribution
• Scaling Laws are Redundancy Laws
• A Neural Scaling Law from the Dimension of the Data Manifold
• Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients
• Scaling Laws for Classical Machine Learning on Tabular Data: A Benchmark Study