2025/10/01 by Hongyi Wang, Hong-Yi Wang, Wang, Hong-Yi +8 · 1 voice · 1 citation
Computer Science · Decision Sciences · Economics, Econometrics and Finance · Mathematics · Physics and Astronomy · #Complex Systems and Time Series Analysis #Stock Market Forecasting Methods #cond-mat.dis-nn #cs.IT #cs.LG #stat.ML
paper · pdf · doi:10.48550/arxiv.2510.00504
arxiv published 2025/10/01 · arxiv updated 2026/03/02
When training large-scale models, the performance typically scales with the number of parameters and the dataset size according to a slow power law. A fundamental theoretical and practical question is whether comparable performance can be achieved with significantly smaller models and substantially less data. In this work, we provide a positive and constructive answer. We prove that a generic permutation-invariant function of d objects can be asymptotically compressed into a function of polylog d objects with vanishing error, which is proved to be the optimal compression rate. This theorem yields two key implications: (Ia) a large neural network can be compressed to polylogarithmic width while preserving its learning dynamics; (Ib) a large dataset can be compressed to polylogarithmic size while leaving the loss landscape of the corresponding model unchanged. Implication (Ia) directly establishes a proof of the dynamical lottery ticket hypothesis, which states that any ordinary network can be strongly compressed such that the learning dynamics and result remain unchanged. (Ib) shows that a neural scaling law of the form L∼ d-α can be boosted to an arbitrarily fast power law decay, and ultimately to exp(-α' √[m]d).