Learning the Stein Discrepancy for Training and Evaluating Energy-Based Models without Sampling

2020/02/13 by Will Grathwohl, Grathwohl, Will, Kuan-Chieh Wang +7
Computer Science · Physics and Astronomy · #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Model Reduction and Neural Networks

paper · pdf · doi:10.48550/arxiv.2002.05616

openalex publication_date 2020/02/13 · openalex created_date 2020/07/02 · openalex updated_date 2026/07/28

Abstract

We present a new method for evaluating and training unnormalized density models. Our approach only requires access to the gradient of the unnormalized model's log-density. We estimate the Stein discrepancy between the data density p(x) and the model density q(x) defined by a vector function of the data. We parameterize this function with a neural network and fit its parameters to maximize the discrepancy. This yields a novel goodness-of-fit test which outperforms existing methods on high dimensional data. Furthermore, optimizing q(x) to minimize this discrepancy produces a novel method for training unnormalized models which scales more gracefully than existing methods. The ability to both learn and compare models is a unique feature of the proposed method.

Citations

• Adam: A Method for Stochastic Optimization
• A Connection Between Score Matching and Denoising Autoencoders
• Training Products of Experts by Minimizing Contrastive Divergence
• A Fast Learning Algorithm for Deep Belief Nets
• A Theory of Generative ConvNet
• Unbiased Markov chain Monte Carlo with couplings
• Searching for Activation Functions
• Spectral Normalization for Generative Adversarial Networks
• Deep Energy Estimator Networks
• Explorations in Homeomorphic Variational Auto-Encoding
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Implicit Generation and Generalization in Energy-Based Models
• Generative Modeling by Estimating Gradients of the Data Distribution
• Flow Contrastive Estimation of Energy-Based Models

Related

• Adversarial Stein Training for Graph Energy Models
• Stein Neural Sampler
• Bridging Explicit and Implicit Deep Generative Models via Neural Stein Estimators
• Neural Stein critics with staged L²-regularization
• On Energy-Based Models with Overparametrized Shallow Neural Networks