Explicit Density Approximation for Neural Implicit Samplers Using a Bernstein-Based Convex Divergence

2025/06/05 by de Frutos, José Manuel, Vázquez, Manuel A., Olmos, Pablo M. +1
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Probability (math.PR)

paper · doi:10.48550/arxiv.2506.04700

Abstract

Rank-based statistical metrics, such as the invariant statistical loss (ISL), have recently emerged as robust and practically effective tools for training implicit generative models. In this work, we introduce dual-ISL, a novel likelihood-free objective for training implicit generative models that interchanges the roles of the target and model distributions in the ISL framework, yielding a convex optimization problem in the space of model densities. We prove that the resulting rank-based discrepancy d_K is i) continuous under weak convergence and with respect to the L¹ norm, and ii) convex in its first argument-properties not shared by classical divergences such as KL or Wasserstein distances. Building on this, we develop a theoretical framework that interprets d_K as an L²-projection of the density ratio q = p/ p onto a Bernstein polynomial basis, from which we derive exact bounds on the truncation error, precise convergence rates, and a closed-form expression for the truncated density approximation. We further extend our analysis to the multivariate setting via random one-dimensional projections, defining a sliced dual-ISL divergence that retains both convexity and continuity. We empirically show that these theoretical advantages translate into practical ones. Specifically, across several benchmarks dual-ISL converges more rapidly, delivers markedly smoother and more stable training, and more effectively prevents mode collapse than classical ISL and other leading implicit generative methods-while also providing an explicit density approximation.

Citations

• Robust training of implicit generative models for multivariate and heavy-tailed distributions with an invariant statistical loss
• Training Implicit Generative Models via an Invariant Statistical Loss
• MCL-GAN: Generative Adversarial Networks with Multiple Specialized Discriminators
• Pareto GAN: Extending the Representational Power of GANs to Heavy-Tailed Distributions
• Generalized Sliced Wasserstein Distances
• Assessing Generative Models via Precision and Recall
• Adversarial Audio Synthesis
• MMD GAN: Towards Deeper Understanding of Moment Matching Network
• Generative Multi-Adversarial Networks
• Learning in Implicit Generative Models
• Robust benchmarking in noisy environments
• Density Estimation Using Real NVP
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

Related

• Robust training of implicit generative models for multivariate and heavy-tailed distributions with an invariant statistical loss
• Bridging Explicit and Implicit Deep Generative Models via Neural Stein Estimators
• Robust training of implicit generative models for multivariate and heavy-tailed distributions with an invariant statistical loss
• Neural Approximate Sufficient Statistics for Implicit Models
• Implicit MLE: Backpropagating Through Discrete Exponential Family Distributions