2022/10/12 by Itai Gat, Gat, Itai, Yossi Adi +5 · 2 citations
Computer Science · Mathematics · #Adversarial Robustness in Machine Learning #Applied mathematics #Bounded function #Discrete mathematics #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Generalization #Lipschitz continuity #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Mathematical analysis #Mathematical optimization #Mathematics #Norm (philosophy) #Pure mathematics #Sobolev space #Uniform norm #Upper and lower bounds
paper · pdf · doi:10.48550/arxiv.2210.06143
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2022/10/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Generalization bounds which assess the difference between the true risk and the empirical risk, have been studied extensively. However, to obtain bounds, current techniques use strict assumptions such as a uniformly bounded or a Lipschitz loss function. To avoid these assumptions, in this paper, we follow an alternative approach: we relax uniform bounds assumptions by using on-average bounded loss and on-average bounded gradient norm assumptions. Following this relaxation, we propose a new generalization bound that exploits the contractivity of the log-Sobolev inequalities. These inequalities add an additional loss-gradient norm term to the generalization bound, which is intuitively a surrogate of the model complexity. We apply the proposed bound on Bayesian deep nets and empirically analyze the effect of this new loss-gradient norm term on different neural architectures.