α-GAN by Rényi Cross Entropy

2025/05/20 by Ni Ding, Ding, Ni, Jiaxing Xu +6
Computer Science · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2505.14190

openalex publication_date 2025/05/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/04

Abstract

This paper proposes α-GAN, a generative adversarial network using Rényi measures. The value function is formulated, by Rényi cross entropy, as an expected certainty measure incurred by the discriminator's soft decision as to where the sample is from, true population or the generator. The discriminator tries to maximize the Rényi certainty about sample source, while the generator wants to reduce it by injecting fake samples. This forms a min-max problem with the solution parameterized by the Rényi order α. This α-GAN reduces to vanilla GAN at α= 1, where the value function is exactly the binary cross entropy. The optimization of α-GAN is over probability (vector) space. It is shown that the gradient is exponentially enlarged when Rényi order is in the range α∈ (0,1). This makes convergence faster, which is verified by experimental results. A discussion shows that choosing α∈ (0,1) may be able to solve some common problems, e.g., vanishing gradient. A following observation reveals that this range has not been fully explored in the existing Rényi version GANs.

Citations

• Addressing GAN Training Instabilities via Tunable Classification Losses
• A Unifying Generator Loss Function for Generative Adversarial Networks
• α-GAN: Convergence and Estimation Guarantees
• StyleSwin: Transformer-based GAN for High-resolution Image Generation
• Realizing GANs via a Tunable Loss Function
• Text to Image Generation with Semantic-Spatial Aware GAN
• A Tunable Loss Function for Robust Classification: Calibration, Landscape, and Generalization
• A Tunable Loss Function for Binary Classification
• Tunable Measures for Information Leakage and Applications to Privacy-Utility Tradeoffs
• Maximum Causal Tsallis Entropy Imitation Learning
• Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels
• Improved Training of Wasserstein GANs
• f-GAN: Training Generative Neural Samplers using Variational Divergence\n Minimization
• Adaptive Restart for Accelerated Gradient Schemes
• Composite Binary Losses

Related

• Improving GAN Training via Binarized Representation Entropy (BRE)\n Regularization
• GANs with Variational Entropy Regularizers: Applications in Mitigating the Mode-Collapse Issue
• A Unifying Generator Loss Function for Generative Adversarial Networks
• Deconstructing Generative Adversarial Networks
• Look ATME: The Discriminator Mean Entropy Needs Attention