Scaling laws for learning with real and surrogate data

2024/02/06 by Ayush Jain, Andrea Montanari, Jain, Ayush +4 · 1 voice · 7 citations
Computer Science · Mathematics · #Artificial intelligence #Computer science #Econometrics #Economics #Gaussian Processes and Bayesian Inference #Machine Learning and Algorithms #Machine learning #Mathematics #Neural Networks and Applications #Scaling #Scaling law #Surrogate model #cs.AI #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2402.04376

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/02/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Collecting large quantities of high-quality data can be prohibitively expensive or impractical, and a bottleneck in machine learning. One may instead augment a small set of n data points from the target distribution with data from more accessible sources, e.g. data collected under different circumstances or synthesized by generative models. We refer to such data as `surrogate data'. We study a weighted empirical risk minimization (ERM) approach for integrating surrogate data into training. We analyze mathematically this method under several classical statistical models, and validate our findings empirically on datasets from different domains. Our main findings are: (i) Integrating surrogate data can significantly reduce the test error on the original distribution. Surprisingly, this can happen even when the surrogate data is unrelated to the original ones. We trace back this behavior to the classical Stein's paradox. (ii) In order to reap the benefit of surrogate data, it is crucial to use optimally weighted ERM. (iii) The test error of models trained on mixtures of real and surrogate data is approximately described by a scaling law. This scaling law can be used to predict the optimal weighting scheme, and to choose the amount of surrogate data to add.

Cited by

• Data Value in the Age of Scaling: Understanding LLM Scaling Dynamics Under Real-Synthetic Data Mixtures
• Optimal Regularization for Performative Learning
• Beyond Real Data: Synthetic Data through the Lens of Regularization
• High-dimensional Analysis of Synthetic Data Selection
• Filtering with Confidence: When Data Augmentation Meets Conformal Prediction
• When Models Don't Collapse: On the Consistency of Iterative MLE
• Residual Feature Integration is Sufficient to Prevent Negative Transfer

Discussions

• If at NeurIPS on Friday, consider stopping by Eren Sasoglu's poster on 'Scaling laws for learning with real and surrogate data' arxiv.org/abs/2402.04376 Often training on a mixture of data from the ta [bsky, 33 points, 2 comments]

Related

• Weighted Empirical Risk Minimization: Sample Selection Bias Correction based on Importance Sampling
• Surrogate-Powered Inference: Regularization and Adaptivity
• Scaling Laws for Optimal Data Mixtures
• Testing for Overfitting
• Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate