vix.ing · top · new · best · stats

Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think

2024/10/09 by Sihyun Yu, Yu, Sihyun, Sangkyung Kwak +11 · 2 voices · 268 citations
Computer Science · Engineering · #Artificial intelligence #Computer science #Electrical engineering #Engineering #Physics #Political science #Politics #Reinforcement Learning in Robotics #Representation (politics) #Training (meteorology) #Transformer

paper · pdf · doi:10.48550/arxiv.2410.06940

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/10/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Recent studies have shown that the denoising process in (generative) diffusion models can induce meaningful (discriminative) representations inside the model, though the quality of these representations still lags behind those learned through recent self-supervised learning methods. We argue that one main bottleneck in training large-scale diffusion models for generation lies in effectively learning these representations. Moreover, training can be made easier by incorporating high-quality external visual representations, rather than relying solely on the diffusion models to learn them independently. We study this by introducing a straightforward regularization called REPresentation Alignment (REPA), which aligns the projections of noisy input hidden states in denoising networks with clean image representations obtained from external, pretrained visual encoders. The results are striking: our simple strategy yields significant improvements in both training efficiency and generation quality when applied to popular diffusion and flow-based transformers, such as DiTs and SiTs. For instance, our method can speed up SiT training by over 17.5×, matching the performance (without classifier-free guidance) of a SiT-XL model trained for 7M steps in less than 400K steps. In terms of final generation quality, our approach achieves state-of-the-art results of FID=1.42 using classifier-free guidance with the guidance interval.

Cited by

Discussions

Related