2025/04/08 by Shuai Wang, Zhi Tian, Wang, Shuai +5 · 1 voice · 84 citations
Computer Science · Engineering · Medicine · #Advanced Memory and Neural Computing #Advanced Neuroimaging Techniques and Applications #Generative Adversarial Networks and Image Synthesis #cs.AI #cs.CV
paper · pdf · doi:10.48550/arxiv.2504.05741
openalex publication_date 2025/04/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Diffusion transformers have demonstrated remarkable generation quality, albeit requiring longer training iterations and numerous inference steps. In each denoising step, diffusion transformers encode the noisy inputs to extract the lower-frequency semantic component and then decode the higher frequency with identical modules. This scheme creates an inherent optimization dilemma: encoding low-frequency semantics necessitates reducing high-frequency components, creating tension between semantic encoding and high-frequency decoding. To resolve this challenge, we propose a new \textbf\colorddtDecoupled \textbf\colorddtDiffusion \textbf\colorddtTransformer~(\textbf\colorddtDDT), with a decoupled design of a dedicated condition encoder for semantic extraction alongside a specialized velocity decoder. Our experiments reveal that a more substantial encoder yields performance improvements as model size increases. For ImageNet 256×256, Our DDT-XL/2 achieves a new state-of-the-art performance of 1.31 FID~(nearly 4× faster training convergence compared to previous diffusion transformers). For ImageNet 512×512, Our DDT-XL/2 achieves a new state-of-the-art FID of 1.28. Additionally, as a beneficial by-product, our decoupled architecture enhances inference speed by enabling the sharing self-condition between adjacent denoising steps. To minimize performance degradation, we propose a novel statistical dynamic programming approach to identify optimal sharing strategies.