Information-Theoretic Bounds on the Moments of the Generalization Error of Learning Algorithms

2021/02/03 by Aminian, Gholamali, Toni, Laura, Rodrigues, Miguel R. D. · 1 citation
#FOS: Computer and information sciences #Information Theory (cs.IT) #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.2102.02016

Abstract

Generalization error bounds are critical to understanding the performance of machine learning models. In this work, building upon a new bound of the expected value of an arbitrary function of the population and empirical risk of a learning algorithm, we offer a more refined analysis of the generalization behaviour of a machine learning models based on a characterization of (bounds) to their generalization error moments. We discuss how the proposed bounds -- which also encompass new bounds to the expected generalization error -- relate to existing bounds in the literature. We also discuss how the proposed generalization error moment bounds can be used to construct new generalization error high-probability bounds.

Cited by

• An Information-Theoretic Analysis of Out-of-Distribution Generalization in Meta-Learning with Applications to Meta-RL

Related

• Generalization Error Bounds via mth Central Moments of the Information Density
• Jensen-Shannon Information Based Characterization of the Generalization Error of Learning Algorithms
• The Generalization Error of Machine Learning Algorithms
• Learning Algorithm Generalization Error Bounds via Auxiliary Distributions
• Information-theoretic analysis of generalization capability of learning algorithms