Are GANs Created Equal? A Large-Scale Study

2017/11/28 by Mario Lucic, Mario Lučić, Karol Kurach +8 · 2 voices · 36 citations
Computer Science · Mathematics · #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Music and Audio Processing #Video Analysis and Summarization #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1711.10337

NIPS'18: Added a section on the limitations of the study and additional empirical results

openalex publication_date 2017/11/28 · arxiv published 2017/11/28 · arxiv created 2018/10/29 · arxiv updated 2018/10/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Generative adversarial networks (GAN) are a powerful subclass of generative models. Despite a very rich research activity leading to numerous interesting GAN algorithms, it is still very hard to assess which algorithm(s) perform better than others. We conduct a neutral, multi-faceted large-scale empirical study on state-of-the art models and evaluation measures. We find that most models can reach similar scores with enough hyperparameter optimization and random restarts. This suggests that improvements can arise from a higher computational budget and tuning more than fundamental algorithmic changes. To overcome some limitations of the current metrics, we also propose several data sets on which precision and recall can be computed. Our experimental results suggest that future GAN research should be based on more systematic and objective evaluation procedures. Finally, we did not find evidence that any of the tested algorithms consistently outperforms the non-saturating GAN introduced in \citegoodfellow2014generative.

Cited by

• LayerEdit: Disentangled Multi-Object Editing via Conflict-Aware Multi-Layer Learning
• Knowledge-based anomaly detection for identifying network-induced shape artifacts
• Limited-Memory LRSGA: An Iterative Method for Computing Nash Equilibria in Competitive Optimization Problems
• A Comprehensive Evaluation Framework for Synthetic Trip Data Generation in Public Transport
• IB-GAN: Disentangled Representation Learning with Information Bottleneck Generative Adversarial Networks
• One Pass Is Not Enough: Recursive Latent Refinement for Generative Models
• MorphGen: Controllable and Morphologically Plausible Generative Cell-Imaging
• Scalable GANs with Transformers
• Comparative Analysis of GAN and Diffusion for MRI-to-CT translation
• Improving the color accuracy of lighting estimation models
• Generative Consistency Models for Estimation of Kinetic Parametric Image Posteriors in Total-Body PET
• Generative Feature Replay For Class-Incremental Learning
• Improved Sub-Visible Particle Classification in Flow Imaging Microscopy via Generative AI-Based Image Synthesis
• Harnessing Diffusion-Yielded Score Priors for Image Restoration
• Diffusion models for multivariate subsurface generation and efficient probabilistic inversion
• On Accurate Evaluation of GANs for Language Generation
• Flows for simultaneous manifold learning and density estimation
• Inducing Hierarchical Compositional Model by Sparsifying Generator Network
• Hybrid Adversarial Spectral Loss Conditional Generative Adversarial Networks for Signal Data Augmentation in Ultra-precision Machining Surface Roughness Prediction
• Reducing Variability of Multiple Instance Learning Methods for Digital Pathology
• Expressive Score-Based Priors for Distribution Matching with Geometry-Preserving Regularization
• Vision Transformers for Multi-Variable Climate Downscaling: Emulating Regional Climate Models with a Shared Encoder and Multi-Decoder Architecture
• Relational Mimic for Visual Adversarial Imitation Learning
• EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning
• Dist-GAN: An Improved GAN using Distance Constraints
• FPAN: Mitigating Replication in Diffusion Models through the Fine-Grained Probabilistic Addition of Noise to Token Embeddings
• STEB: In Search of the Best Evaluation Approach for Synthetic Time Series
• BAGAN: Data Augmentation with Balancing GAN
• Exploring In-Image Machine Translation with Real-World Background
• RSRA: Training-Free Probing of Representation Sensitivity for Efficient LoRA Rank Allocation
• Sample-Efficient Imitation Learning via Generative Adversarial Nets
• Ensuring Reproducibility in Generative AI Systems for General Use Cases: A Framework for Regression Testing and Open Datasets
• Can Training Logs Make Model Comparisons More Precise?
• A Style-Based Generator Architecture for Generative Adversarial Networks
• The Devil is in the Decoder: Classification, Regression and GANs
• Diffusion Models for Robotic Manipulation: A Survey
• Generative Adversarial Networks with Limited Data: A Survey and Benchmarking

Discussions

• Are GANs Created Equal? A Large-Scale Study [hn, 3 points, 0 comments]
• Are GANs Created Equal? A Large-Scale Study (2017) [hn, 2 points, 0 comments]

Related

• Pros and Cons of GAN Evaluation Measures
• An empirical study on evaluation metrics of generative adversarial networks
• On the Performance of Generative Adversarial Network (GAN) Variants: A Clinical Data Study
• How good is my GAN?
• On Accurate Evaluation of GANs for Language Generation