Image Classification Using a Diffusion Model as a Pre-Training Model

2025/05/11 by Ukita, Kosuke, Xiaolong, Ye, Okita, Tsuyoshi · 1 citation
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #FOS: Electrical engineering #Image and Video Processing (eess.IV) #Machine Learning (cs.LG) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2505.06890

Abstract

In this paper, we propose a diffusion model that integrates a representation-conditioning mechanism, where the representations derived from a Vision Transformer (ViT) are used to condition the internal process of a Transformer-based diffusion model. This approach enables representation-conditioned data generation, addressing the challenge of requiring large-scale labeled datasets by leveraging self-supervised learning on unlabeled data. We evaluate our method through a zero-shot classification task for hematoma detection in brain imaging. Compared to the strong contrastive learning baseline, DINOv2, our method achieves a notable improvement of +6.15% in accuracy and +13.60% in F1-score, demonstrating its effectiveness in image classification.

Citations

• Diffusion Models Beat GANs on Image Classification
• Your Diffusion Model is Secretly a Zero-Shot Classifier
• Denoising Diffusion Autoencoders are Unified Self-supervised Learners
• High Fidelity Visualization of What Your Self-Supervised Representation Knows About
• Vector Quantized Diffusion Model for Text-to-Image Synthesis
• Diffusion Models Beat GANs on Image Synthesis
• Emerging Properties in Self-Supervised Vision Transformers
• Classification of Hematoma: Joint Learning of Semantic Segmentation and Classification
• Generating Images with Sparse Representations
• Taming Transformers for High-Resolution Image Synthesis
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Contrastive Learning with Hard Negative Samples
• Denoising Diffusion Probabilistic Models
• A Simple Framework for Contrastive Learning of Visual Representations
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Momentum Contrast for Unsupervised Visual Representation Learning
• Momentum Contrast for Unsupervised Visual Representation Learning
• Improved Precision and Recall Metric for Assessing Generative Models
• Attention Is All You Need
• Improved Techniques for Training GANs
• Deep Residual Learning for Image Recognition
• Rethinking the Inception Architecture for Computer Vision

Cited by

• From Understanding to Erasing: Towards Complete and Stable Video Object Removal

Related

• An Efficient Framework for Enhancing Discriminative Models via Diffusion Techniques
• Training on Thin Air: Improve Image Classification with Generated Data
• Denoising Diffusion Autoencoders are Unified Self-supervised Learners
• Diffusion Models Beat GANs on Image Classification
• Your Diffusion Model is Secretly a Zero-Shot Classifier