Every Model Learned by Gradient Descent Is Approximately a Kernel Machine

2020/11/30 by Pedro Domingos, Domingos, Pedro · 11 voices · 1 citation
Computer Science · #Adversarial Robustness in Machine Learning #Domain Adaptation and Few-Shot Learning #Generative Adversarial Networks and Image Synthesis #cs.LG #cs.NE #stat.ML

paper · pdf · doi:10.48550/arxiv.2012.00152

openalex publication_date 2020/11/30 · openalex created_date 2020/12/07 · openalex updated_date 2026/07/28

Abstract

Deep learning's successes are often attributed to its ability to automatically discover new representations of the data, rather than relying on handcrafted features like other learning methods. We show, however, that deep networks learned by the standard gradient descent algorithm are in fact mathematically approximately equivalent to kernel machines, a learning method that simply memorizes the data and uses it directly for prediction via a similarity function (the kernel). This greatly enhances the interpretability of deep network weights, by elucidating that they are effectively a superposition of the training examples. The network architecture incorporates knowledge of the target function into the kernel. This improved understanding should lead to better learning algorithms.

Citations

• A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting
• Gradient-based learning applied to document recognition
• On Estimation of a Probability Density Function and Mode
• Representation Learning: A Review and New Perspectives

Cited by

• Instruction-Level Weight Shaping: A Framework for Self-Improving AI Agents

Discussions

• Every Model Learned by Gradient Descent Is Approximately a Kernel Machine [hn, 406 points, 107 comments]
• Every model learned by gradient descent is approximately a kernel machine (2020) [hn, 176 points, 136 comments]
• Every Model Learned by Gradient Descent Is Approximately a Kernel Machine [hn, 4 points, 0 comments]
• Every Model Learned by Gradient Descent Is Approximately a Kernel Machine [hn, 4 points, 0 comments]
• from the folder: "papers that make something click" arxiv.org/pdf/2012.00152 [bsky, 3 points, 1 comments]
• Deep learning networks are approximately kernel machines [hn, 2 points, 1 comments]
• Every Model Learned by Gradient Descent in Approximately a Colonel Machine …just one step down from General AI… 🥁 Sorry, I’ll see myself out. (*) paper does seem kinda cool. arxiv.org/abs/2012.00152 [bsky, 2 points, 0 comments]
• Every Model Learned By Gradient Descent Is Approximately A Kernel Machine (2020) [hn, 2 points, 0 comments]
• Omg this paper if true "deep network weights, by elucidating that they are effectively a superposition of the training examples" #DeepLearning #generativeAI arxiv.org/abs/2012.001... [bsky, 2 points, 0 comments]
• This is very interesting and possibly effective repellant for the current AGI hype: arxiv.org/abs/2012.0... [bsky, 0 points, 0 comments]
• Every Model Learned by Gradient Descent Is Approximately a Kernel Machine (arxiv.org) Main Link | Discussion [bsky, 0 points, 0 comments]

Related

• Convolutional Deep Kernel Machines
• Learning Explicit Deep Representations from Deep Kernel Networks
• Can neural networks extrapolate? Discussion of a theorem by Pedro Domingos
• Neural Networks can Learn Representations with Gradient Descent
• Mechanism of feature learning in deep fully connected networks and kernel machines that recursively learn features