2024/10/06 by A. E. Beck, Alon Beck, Beck, Alon +4 · 4 voices · 4 citations
Computer Science · Mathematics · Physics and Astronomy · #Artificial intelligence #Computer science #Enhanced Data Rates for GSM Evolution #Mathematics #Statistical Mechanics and Entropy #Stochastic Gradient Optimization Techniques #Theoretical and Computational Physics
paper · pdf · doi:10.48550/arxiv.2410.04489
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2024/10/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
We study the generalization properties of binary logistic classification in a simplified setting, for which a "memorizing" and "generalizing" solution can always be strictly defined, and elucidate empirically and analytically the mechanism underlying Grokking in its dynamics. We analyze the asymptotic long-time dynamics of logistic classification on a random feature model with a constant label and show that it exhibits Grokking, in the sense of delayed generalization and non-monotonic test loss. We find that Grokking is amplified when classification is applied to training sets which are on the verge of linear separability. Even though a perfect generalizing solution always exists, we prove the implicit bias of the logisitc loss will cause the model to overfit if the training data is linearly separable from the origin. For training sets that are not separable from the origin, the model will always generalize perfectly asymptotically, but overfitting may occur at early stages of training. Importantly, in the vicinity of the transition, that is, for training sets that are almost separable from the origin, the model may overfit for arbitrarily long times before generalizing. We gain more insights by examining a tractable one-dimensional toy model that quantitatively captures the key features of the full model. Finally, we highlight intriguing common properties of our findings with recent literature, suggesting that grokking generally occurs in proximity to the interpolation threshold, reminiscent of critical phenomena often observed in physical systems.