Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyond

2024/10/31 by Alan Jeffares, Alicia Curth, Jeffares, Alan +3 · 6 voices · 1 citation
Computer Science · Engineering · #Architecture and Computational Design #Neural Networks and Reservoir Computing #Stochastic Gradient Optimization Techniques #cs.AI #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2411.00247

openalex publication_date 2024/10/31 · openalex created_date 2024/11/14 · openalex updated_date 2026/07/28

Abstract

Deep learning sometimes appears to work in unexpected ways. In pursuit of a deeper understanding of its surprising behaviors, we investigate the utility of a simple yet accurate model of a trained neural network consisting of a sequence of first-order approximations telescoping out into a single empirically operational tool for practical analysis. Across three case studies, we illustrate how it can be applied to derive new empirical insights on a diverse range of prominent phenomena in the literature -- including double descent, grokking, linear mode connectivity, and the challenges of applying deep learning on tabular data -- highlighting that this model allows us to construct and extract metrics that help predict and understand the a priori unexpected performance of neural networks. We also demonstrate that this model presents a pedagogical formalism allowing us to isolate components of the training process even in complex contemporary settings, providing a lens to reason about the effects of design choices such as architecture & optimization strategy, and reveals surprising parallels between neural network learning and gradient boosting.

Cited by

• Not All Explanations for Deep Learning Phenomena Are Equally Valuable

Discussions

• In the meantime, check out our paper for more: arxiv.org/abs/2411.00247 but I’ll be back with more soon!! 🫳🎤20/20 [bsky, 12 points, 2 comments]
• *Deep Learning Through A Telescoping Lens* by @alanjeffares.bsky.social @aliciacurth.bsky.social Shows that tracking 1st-order approximations to the training dynamics provides insights into many phen [bsky, 10 points, 0 comments]
• This is a cool paper arxiv.org/abs/2411.00247, do recommend! [bsky, 7 points, 1 comments]
• In our new "Advanced Methods for Machine Learning" module, this week seminar dug into an upcoming NeurIPS paper arxiv.org/abs/2411.00247 that provides a new tool for analysing changes in effective mod [bsky, 2 points, 1 comments]
• Agreed, but if we use neural nets for outcome nuisance in Double ML, we can at most hope for approximate weights following arxiv.org/abs/2411.00247 If you use (penalized) logistic regression instead, [bsky, 1 points, 1 comments]
• "highlighting that this model allows us to construct and extract metrics that help predict and understand the a priori unexpected performance of neural networks." [bsky, 1 points, 0 comments]

Related

• A Selective Overview of Deep Learning
• Deep Grokking: Would Deep Neural Networks Generalize Better?
• Not All Explanations for Deep Learning Phenomena Are Equally Valuable
• Grokking Beyond Neural Networks: An Empirical Exploration with Model Complexity
• Neural networks: from the perceptron to deep nets