Deep Learning is Not So Mysterious or Different

2025/03/03 by Andrew Gordon Wilson, Wilson, Andrew Gordon · 31 voices · 26 citations
Computer Science · #Explainable Artificial Intelligence (XAI) #Generative Adversarial Networks and Image Synthesis #Stochastic Gradient Optimization Techniques #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2503.02113

openalex publication_date 2025/03/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Deep neural networks are often seen as different from other model classes by defying conventional notions of generalization. Popular examples of anomalous generalization behaviour include benign overfitting, double descent, and the success of overparametrization. We argue that these phenomena are not distinct to neural networks, or particularly mysterious. Moreover, this generalization behaviour can be intuitively understood, and rigorously characterized, using long-standing generalization frameworks such as PAC-Bayes and countable hypothesis bounds. We present soft inductive biases as a key unifying principle in explaining these phenomena: rather than restricting the hypothesis space to avoid overfitting, embrace a flexible hypothesis space, with a soft preference for simpler solutions that are consistent with the data. This principle can be encoded in many model classes, and thus deep learning is not as mysterious or different from other model classes as it might seem. However, we also highlight how deep learning is relatively distinct in other ways, such as its ability for representation learning, phenomena such as mode connectivity, and its relative universality.

Citations

• SOAP: Improving and Stabilizing Shampoo using Adam
• Unlocking Tokens as Data Points for Generalization Bounds on Larger Language Models
• Large Language Models Are Zero-Shot Time Series Forecasters
• Training Compute-Optimal Large Language Models
• User-friendly Introduction to PAC-Bayes Bounds
• Understanding deep learning (still) requires rethinking generalization
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Understanding deep learning requires rethinking generalization
• ImageNet classification with deep convolutional neural networks

Cited by

• Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data
• Not-So-Strange Love: Language Models and Generative Linguistic Theories are More Compatible than They Appear
• Across the Levels of Analysis: Explaining Predictive Processing in Humans Requires More Than Machine-Estimated Probabilities
• There Will Be a Scientific Theory of Deep Learning
• You can’t fight in here! This is BBS!
• Linguists should learn to love speech-based deep learning models
• Flexible inference for animal learning rules using neural networks
• How linguistics learned to stop worrying and love the language models
• A PAC-Bayesian approach to generalization for quantum models
• Semantic Space Search Trajectory Networks
• Block-Recurrent Dynamics in Vision Transformers
• Soft Geometric Inductive Bias for Object Centric Dynamics
• Large language models are not about natural language
• Soft Inductive Bias Approach via Explicit Reasoning Perspectives in Inappropriate Utterance Detection Using Large Language Models
• Uncertainty-Aware Data-Efficient AI: An Information-Theoretic Perspective
• Parabolic Position Encoding: Vision-Centric, Principled, Extrapolatable, General
• DNNs, Dataset Statistics, and Correlation Functions
• Sharp Minima Can Generalize: A Loss Landscape Perspective On Data
• A Compression Perspective on Simplicity Bias
• The False Promise of Zero-Shot Super-Resolution in Machine-Learned Operators
• Understanding Transformers for Time Series: Rank Structure, Flow-of-ranks, and Compressibility
• Response to Promises and Pitfalls of Deep Kernel Learning
• Are Language Models Models?
• Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation
• FastNet: Improving the physical consistency of machine-learning weather prediction models through loss function design
• Knowledge distillation as a pathway toward next-generation intelligent ecohydrological modeling systems
• Double descent [wikipedia]

Discussions

• Deep Learning Is Not So Mysterious or Different [hn, 485 points, 126 comments]
• My new paper "Deep Learning is Not So Mysterious or Different": arxiv.org/abs/2503.02113. Generalization behaviours in deep learning can be intuitively understood through a notion of soft inductive bi [bsky, 209 points, 6 comments]
• Deep Learning is Not So Mysterious or Different - arxiv.org/abs/2503.02113 Section 8, listing properties unique to neural networks, is quite interesting. [bsky, 5 points, 0 comments]
• This is a good read ! Paul I think you'd enjoy this paper as well arxiv.org/pdf/2503.02113. You touch that in the conclusion but something is to be written in a course on how to connect simple NN one [bsky, 2 points, 1 comments]
• Funny enough, I later found my perspective on soft constraints to be quite similar as that of soft inductive biases by @andrewgwils.bsky.social in one of his latest works: arxiv.org/abs/2503.02113 🧵3 [bsky, 2 points, 1 comments]
• Deep Learning Is Not So Mysterious or Different [hn, 1 points, 0 comments]
• 8/ Deep learning isn’t magic - it just builds on long-standing ideas in machine learning. Understanding why it works will help us make better models instead of getting caught up in supposed mysteries. [bsky, 1 points, 0 comments]
• Double descent is an artifact of "benign overfitting" and is not specific to deep neural networks. arxiv.org/pdf/2503.02113 [bsky, 0 points, 0 comments]
• "rather than restricting the hypothesis space to avoid overfitting, embrace a flexible hypothesis space, with a soft preference for simpler solutions that are consistent with the data" arxiv.org/abs/2 [bsky, 0 points, 0 comments]
• "Deep Learning Is Not So Mysterious or Different" Deep learning is simpler than it seems. With the right resources, anyone can learn about it and understand complex ideas clearly. Article | Discussio [bsky, 0 points, 0 comments]
• Deep Learning Is Not So Mysterious or Different [bsky, 0 points, 0 comments]
• Deep Learning Is Not So Mysterious or Different #HackerNews https://arxiv.org/abs/2503.02113 [bsky, 0 points, 0 comments]
• Deep Learning Is Not So Mysterious or Different https://arxiv.org/abs/2503.02113 https://news.ycombinator.com/item?id=43390400 [bsky, 0 points, 0 comments]
• Deep Learning Is Not So Mysterious or Different https://arxiv.org/abs/2503.02113 arxiv.org [bsky, 0 points, 0 comments]
• Deep Learning Is Not So Mysterious or Different https://arxiv.org/abs/2503.02113 [bsky, 0 points, 0 comments]
• Deep Learning Is Not So Mysterious or Different https://arxiv.org/abs/2503.02113 [comments] [293 points] [bsky, 0 points, 0 comments]
• New pre-print: "FOUNDATION MODELS FOR SEISMIC DATA PROCESSING: AN EXTENSIVE REVIEW". Interesting findings for #Seismic / #Geophysic Applications: I) hierarchical models outperform basic approaches, II [bsky, 0 points, 0 comments]
• Глубинное обучение не так уж загадочно и не так уж отличается #ai #deeplearning #news [bsky, 0 points, 0 comments]
• Deep Learning Is Not So Mysterious or Different #ai #deeplearning #news [bsky, 0 points, 0 comments]
• arxiv.org/pdf/2503.02113 Courtesy of Andrew Gordon Wilson in the other place. If he has a handle here please send it over. [bsky, 0 points, 0 comments]
• Deep Learning Is Not So Mysterious or Different https://arxiv.org/abs/2503.02113 [bsky, 0 points, 0 comments]
• ⚡ Hackernews Top story: Deep Learning Is Not So Mysterious or Different [bsky, 0 points, 0 comments]
• https://bsky.app/profile/news.ycombinator.com.web.brid.gy/post/3lkn2mfcz5sk2 [bsky, 0 points, 0 comments]
• Deep Learning Is Not So Mysterious or Different https://arxiv.org/abs/2503.02113 (http://news.ycombinator.com/item?id=43390400) [bsky, 0 points, 0 comments]
• Deep Learning Is Not So Mysterious or Different https://arxiv.org/abs/2503.02113 (http://news.ycombinator.com/item?id=43390400) [bsky, 0 points, 0 comments]
• Deep Learning Is Not So Mysterious or Different (arxiv.org) Main Link | Discussion [bsky, 0 points, 0 comments]
• "In other words, understanding deep learning does not require re-thinking generalization, and it never did." I'm not entirely convinced - but interesting to see a 2025 response - arxiv.org/abs/2503.0 [bsky, 0 points, 0 comments]
• Deep Learning Is Not So Mysterious or Different view on hacker news [bsky, 0 points, 0 comments]
• Deep Learning Is Not So Mysterious or Different https://arxiv.org/abs/2503.02113 (https://news.ycombinator.com/item?id=43390400) [bsky, 0 points, 0 comments]
• Deep Learning Is Not So Mysterious or Different https://arxiv.org/abs/2503.02113 (https://news.ycombinator.com/item?id=43390400) [bsky, 0 points, 0 comments]
• Deep Learning Is Not So Mysterious or Different https://arxiv.org/abs/2503.02113 (https://news.ycombinator.com/item?id=43390400) [bsky, 0 points, 0 comments]

Related

• On the Generalization Mystery in Deep Learning
• Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior
• Deep learning generalizes because the parameter-function map is biased towards simple functions
• The Low-Rank Simplicity Bias in Deep Networks
• Exploring Generalization in Deep Learning