Generalization in Machine Learning via Analytical Learning Theory

2018/02/21 by Kenji Kawaguchi, Kawaguchi, Kenji, Yoshua Bengio +5 · 1 citation
Computer Science · Mathematics · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Neural Networks and Applications #Neural and Evolutionary Computing (cs.NE) #cs.AI #cs.LG #cs.NE #stat.ML

paper · pdf · doi:10.48550/arxiv.1802.07426

openalex publication_date 2018/02/21 · arxiv created 2019/03/06 · arxiv updated 2019/03/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper introduces a novel measure-theoretic theory for machine learning that does not require statistical assumptions. Based on this theory, a new regularization method in deep learning is derived and shown to outperform previous methods in CIFAR-10, CIFAR-100, and SVHN. Moreover, the proposed theory provides a theoretical basis for a family of practically successful regularization methods in deep learning. We discuss several consequences of our results on one-shot learning, representation learning, deep learning, and curriculum learning. Unlike statistical learning theory, the proposed learning theory analyzes each problem instance individually via measure theory, rather than a set of problem instances via statistics. As a result, it provides different types of results and insights when compared to statistical learning theory.

Citations

• The Loss Surfaces of Multilayer Networks
• Bayesian Optimization with Exponential Convergence
• Wide Residual Networks
• Sharp Minima Can Generalize For Deep Nets
• Improved Regularization of Convolutional Neural Networks with Cutout
• Deep Learning Scaling is Predictable, Empirically
• Theory of Deep Learning III: explaining the non-overfitting puzzle

Cited by

• Semi-Supervised Learning by Disentangling and Self-Ensembling Over Stochastic Latent Space

Related

• On Generalization and Regularization in Deep Learning
• Implicit Regularization in Deep Learning
• Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitate
• Exploring Generalization in Deep Learning
• Generalizing in the Real World with Representation Learning