2020/10/13 by Sharu Theresa Jose, Osvaldo Simeone, Jose, Sharu Theresa +1 · 2 citations
Computer Science · Engineering · Mathematics · #Bounded function #Combinatorics #Computer science #Discrete mathematics #Distribution (mathematics) #Divergence (linguistics) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #FOS: Electrical engineering #Gaussian Processes and Bayesian Inference #Generalization #Information Theory (cs.IT) #Kullback–Leibler divergence #Machine Learning (cs.LG) #Machine Learning and Algorithms #Mathematical analysis #Mathematics #Mutual information #Signal Processing (eess.SP) #Statistics #Transfer (computing) #Upper and lower bounds #cs.IT #cs.LG #eess.SP #electronic engineering #information engineering #math.IT
paper · pdf · doi:10.48550/arxiv.2010.09484
published in arXiv (Cornell University) (Cornell University) · Submitted for conference publication
openalex publication_date 2020/10/13 · arxiv created 2021/01/25 · arxiv updated 2021/01/26 · openalex created_date 2022/07/25 · openalex updated_date 2026/08/06
In transfer learning, training and testing data sets are drawn from different data distributions. The transfer generalization gap is the difference between the population loss on the target data distribution and the training loss. The training data set generally includes data drawn from both source and target distributions. This work presents novel information-theoretic upper bounds on the average transfer generalization gap that capture (i) the domain shift between the target data distribution P'Z and the source distribution PZ through a two-parameter family of generalized (α1,α2)-Jensen-Shannon (JS) divergences; and (ii) the sensitivity of the transfer learner output W to each individual sample of the data set Zi via the mutual information I(W;Zi). For α1 ∈ (0,1), the (α1,α2)-JS divergence can be bounded even when the support of PZ is not included in that of P'Z. This contrasts the Kullback-Leibler (KL) divergence DKL(PZ||P'Z)-based bounds of Wu et al. [1], which are vacuous under this assumption. Moreover, the obtained bounds hold for unbounded loss functions with bounded cumulant generating functions, unlike the ϕ-divergence based bound of Wu et al. [1]. We also obtain new upper bounds on the average transfer excess risk in terms of the (α1,α2)-JS divergence for empirical weighted risk minimization (EWRM), which minimizes the weighted average training losses over source and target data sets. Finally, we provide a numerical example to illustrate the merits of the introduced bounds.