2023/10/23 by Jiatao Gu, Gu, Jiatao, Shuangfei Zhai +7 · 1 voice · 20 citations
Computer Science · Mathematics · #AI in cancer detection #Advanced Image Processing Techniques #Algorithm #Artificial intelligence #Computer science #Computer vision #Diffusion #Encoder #Generalization #Generative Adversarial Networks and Image Synthesis #Image (mathematics) #Image resolution #Mathematics #Pixel #Process (computing) #Schedule #Space (punctuation)
paper · pdf · doi:10.48550/arxiv.2310.15111
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2023/10/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Diffusion models are the de facto approach for generating high-quality images and videos, but learning high-dimensional models remains a formidable task due to computational and optimization challenges. Existing methods often resort to training cascaded models in pixel space or using a downsampled latent space of a separately trained auto-encoder. In this paper, we introduce Matryoshka Diffusion Models(MDM), an end-to-end framework for high-resolution image and video synthesis. We propose a diffusion process that denoises inputs at multiple resolutions jointly and uses a NestedUNet architecture where features and parameters for small-scale inputs are nested within those of large scales. In addition, MDM enables a progressive training schedule from lower to higher resolutions, which leads to significant improvements in optimization for high-resolution generation. We demonstrate the effectiveness of our approach on various benchmarks, including class-conditioned image generation, high-resolution text-to-image, and text-to-video applications. Remarkably, we can train a single pixel-space model at resolutions of up to 1024x1024 pixels, demonstrating strong zero-shot generalization using the CC12M dataset, which contains only 12 million images. Our code is released at https://github.com/apple/ml-mdm