Representative Feature Extraction During Diffusion Process for Sketch Extraction with One Example

2024/01/09 by Yun, Kwan, Kim, Youngseo, Seo, Kwanggyoon +2 · 3 citations
#68T01 #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Graphics (cs.GR) #I.4.9

paper · doi:10.48550/arxiv.2401.04362

Abstract

We introduce DiffSketch, a method for generating a variety of stylized sketches from images. Our approach focuses on selecting representative features from the rich semantics of deep features within a pretrained diffusion model. This novel sketch generation method can be trained with one manual drawing. Furthermore, efficient sketch extraction is ensured by distilling a trained generator into a streamlined extractor. We select denoising diffusion features through analysis and integrate these selected features with VAE features to produce sketches. Additionally, we propose a sampling scheme for training models using a conditional generative approach. Through a series of comparisons, we verify that distilled DiffSketch not only outperforms existing state-of-the-art sketch extraction methods but also surpasses diffusion-based stylization methods in the task of extracting sketches.

Cited by

• Encrypted Traffic Detection in Resource Constrained IoT Networks: A Diffusion Model and LLM Integrated Framework
• StyleMM: Stylized 3D Morphable Face Model via Text-Driven Aligned Image Translation
• Style Transfer: A Decade Survey

Related

• DiffSketching: Sketch Control Image Synthesis with Diffusion Models
• It's All About Your Sketch: Democratising Sketch Control in Diffusion Models
• d-Sketch: Improving Visual Fidelity of Sketch-to-Image Translation with Pretrained Latent Diffusion Models without Retraining
• DiffSketcher: Text Guided Vector Sketch Synthesis through Latent Diffusion Models
• SketchFFusion: Sketch-guided image editing with diffusion model