2020/03/06 by Lantao Yu, Yang Song, Yu, Lantao +5 · 11 citations
Computer Science · Mathematics · Physics and Astronomy · #Applied mathematics #Artificial intelligence #Computer science #Convergence (economics) #Divergence (linguistics) #FOS: Computer and information sciences #Function (biology) #Gaussian Processes and Bayesian Inference #Generative Adversarial Networks and Image Synthesis #Kullback–Leibler divergence #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Mathematical optimization #Mathematics #Model Reduction and Neural Networks #Parametrization (atmospheric modeling) #Physics #cs.LG #stat.ML
paper · pdf · doi:10.48550/arxiv.2003.03463
published in arXiv (Cornell University) (Cornell University) · ICML 2020
openalex publication_date 2020/03/06 · arxiv created 2020/07/21 · arxiv updated 2020/07/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Deep energy-based models (EBMs) are very flexible in distribution parametrization but computationally challenging because of the intractable partition function. They are typically trained via maximum likelihood, using contrastive divergence to approximate the gradient of the KL divergence between data and model distribution. While KL divergence has many desirable properties, other f-divergences have shown advantages in training implicit density generative models such as generative adversarial networks. In this paper, we propose a general variational framework termed f-EBM to train EBMs using any desired f-divergence. We introduce a corresponding optimization algorithm and prove its local convergence property with non-linear dynamical systems theory. Experimental results demonstrate the superiority of f-EBM over contrastive divergence, as well as the benefits of training EBMs using f-divergences other than KL.