VIVAT: Virtuous Improving VAE Training through Artifact Mitigation

2025/06/09 by Lev Novitskiy, Viacheslav Vasilev, Novitskiy, Lev +7 · 1 citation
Computer Science · #Computer Graphics and Visualization Techniques #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Face recognition and analysis #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Multimedia (cs.MM)

paper · pdf · doi:10.48550/arxiv.2506.07863

openalex publication_date 2025/06/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Variational Autoencoders (VAEs) remain a cornerstone of generative computer vision, yet their training is often plagued by artifacts that degrade reconstruction and generation quality. This paper introduces VIVAT, a systematic approach to mitigating common artifacts in KL-VAE training without requiring radical architectural changes. We present a detailed taxonomy of five prevalent artifacts - color shift, grid patterns, blur, corner and droplet artifacts - and analyze their root causes. Through straightforward modifications, including adjustments to loss weights, padding strategies, and the integration of Spatially Conditional Normalization, we demonstrate significant improvements in VAE performance. Our method achieves state-of-the-art results in image reconstruction metrics (PSNR and SSIM) across multiple benchmarks and enhances text-to-image generation quality, as evidenced by superior CLIP scores. By preserving the simplicity of the KL-VAE framework while addressing its practical challenges, VIVAT offers actionable insights for researchers and practitioners aiming to optimize VAE training.

Citations

• EQ-VAE: Equivariance Regularized Latent Space for Improved Generative Image Modeling
• Cosmos World Foundation Model Platform for Physical AI
• HunyuanVideo: A Systematic Framework For Large Video Generative Models
• Kandinsky 3: Text-to-Image Synthesis for Multifunctional Generative Framework
• Movie Gen: A Cast of Media Foundation Models
• Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models
• CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer
• Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models
• Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
• Fast Timing-Conditioned Latent Audio Diffusion
• SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
• LAION-5B: An open large-scale dataset for training next generation image-text models
• MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generation
• Hierarchical Text-Conditional Image Generation with CLIP Latents
• DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latents
• Vector-quantized Image Modeling with Improved VQGAN
• Learning Transferable Visual Models From Natural Language Supervision
• Focal Frequency Loss for Image Reconstruction and Synthesis
• Taming Transformers for High-Resolution Image Synthesis
• Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images
• NVAE: A Deep Hierarchical Variational Autoencoder
• Simple and Effective VAE Training with Calibrated Decoders
• Understanding and Improving Layer Normalization
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• A Style-Based Generator Architecture for Generative Adversarial Networks
• Variational Autoencoder with Implicit Optimal Priors
• Group Normalization
• The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
• Wasserstein Auto-Encoders
• Neural Discrete Representation Learning
• Attention Is All You Need
• Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders
• Perceptual Losses for Real-Time Style Transfer and Super-Resolution
• Autoencoding beyond pixels using a learned similarity metric
• Deep Residual Learning for Image Recognition
• Attribute2Image: Conditional Image Generation from Visual Attributes
• Microsoft COCO: Common Objects in Context

Cited by

• ART-VITON: Measurement-Guided Latent Diffusion for Artifact-Free Virtual Try-On

Related

• VCE: Variational Convertor-Encoder for One-Shot Generalization
• Optimizing Hierarchical Image VAEs for Sample Quality
• Improve variational autoEncoder with auxiliary softmax multiclassifier
• Energy-Calibrated VAE with Test Time Free Lunch
• Improving Conditional VAE with approximation using Normalizing Flows