2021/12/10 by Matías Vera, Matias Vera, Leonardo Rey Vega +4 · 1 citation
Computer Science · Engineering · Mathematics · #Algorithm #Artificial intelligence #Artificial neural network #Computer science #Cross entropy #FOS: Computer and information sciences #Fault Detection and Control Systems #Function (biology) #Generalization #Generalization error #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Machine Learning and Data Classification #Mathematics #Metric (unit) #Pattern recognition (psychology) #Perspective (graphical) #Set (abstract data type) #cs.LG #stat.ML
paper · pdf · doi:10.48550/arxiv.2112.05547
Submitted to be considered for publication in Information and Inference: a Journal of the IMA
arxiv created 2021/12/10 · openalex publication_date 2021/12/10 · arxiv updated 2021/12/13 · openalex created_date 2021/12/31 · openalex updated_date 2026/08/05
The ultimate performance of machine learning algorithms for classification tasks is usually measured in terms of the empirical error probability (or accuracy) based on a testing dataset. Whereas, these algorithms are optimized through the minimization of a typically different--more convenient--loss function based on a training set. For classification tasks, this loss function is often the negative log-loss that leads to the well-known cross-entropy risk which is typically better behaved (from a numerical perspective) than the error probability. Conventional studies on the generalization error do not usually take into account the underlying mismatch between losses at training and testing phases. In this work, we introduce an analysis based on point-wise PAC approach over the generalization gap considering the mismatch of testing based on the accuracy metric and training on the negative log-loss. We label this analysis PACMAN. Building on the fact that the mentioned mismatch can be written as a likelihood ratio, concentration inequalities can be used to provide some insights for the generalization problem in terms of some point-wise PAC bounds depending on some meaningful information-theoretic quantities. An analysis of the obtained bounds and a comparison with available results in the literature are also provided.