Improved Contrastive Divergence Training of Energy Based Models

2020/12/02 by Yilun Du, Shuang Li, Du, Yilun +5 · 9 citations
Computer Science · Physics and Astronomy · #Generative Adversarial Networks and Image Synthesis #Adversarial Robustness in Machine Learning #Model Reduction and Neural Networks

paper · pdf · doi:10.48550/arxiv.2012.01316

Abstract

Contrastive divergence is a popular method of training energy-based models, but is known to have difficulties with training stability. We propose an adaptation to improve contrastive divergence training by scrutinizing a gradient term that is difficult to calculate and is often left out for convenience. We show that this gradient term is numerically significant and in practice is important to avoid training instabilities, while being tractable to estimate. We further highlight how data augmentation and multi-scale processing can be used to improve model robustness and generation quality. Finally, we empirically evaluate stability of model architectures and show improved performance on a host of benchmarks and use cases,such as image generation, OOD detection, and compositional generation.

Citations

• Pixel Recurrent Neural Networks
• Stein Variational Policy Gradient
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• A Simple Framework for Contrastive Learning of Visual Representations

Cited by

• EBT-Policy: Energy Unlocks Emergent Physical Reasoning Capabilities
• Pre-training Molecular Graph Representation with 3D Geometry
• Distance Marching for Generative Modeling
• Generalizable Reasoning through Compositional Energy Minimization
• Compositional Visual Generation and Inference with Energy Based Models
• SCOPED: Score-Curvature Out-of-distribution Proximity Evaluator for Diffusion
• Learning Latent Energy-Based Models via Interacting Particle Langevin Dynamics
• Equilibrium Matching: Generative Modeling with Implicit Energy-Based Models
• Uniform-in-time convergence bounds for Persistent Contrastive Divergence Algorithms
• EigenScore: OOD Detection using Covariance in Diffusion Models

Related

• Training Energy-Based Models with Diffusion Contrastive Divergences
• Generalized Contrastive Divergence: Joint Training of Energy-Based Model and Diffusion Model through Inverse Reinforcement Learning
• Learning Deep Energy Models: Contrastive Divergence vs. Amortized MLE
• Clarifying MCMC-based training of modern EBMs : Contrastive Divergence versus Maximum Likelihood
• Guiding Energy-based Models via Contrastive Latent Variables