Draw Your Mind: Personalized Generation via Condition-Level Modeling in Text-to-Image Diffusion Models

2025/08/05 by Kim, Hyungjin, Ahn, Seokho, Seo, Young-Duk · 1 citation
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2508.03481

Abstract

Personalized generation in T2I diffusion models aims to naturally incorporate individual user preferences into the generation process with minimal user intervention. However, existing studies primarily rely on prompt-level modeling with large-scale models, often leading to inaccurate personalization due to the limited input token capacity of T2I diffusion models. To address these limitations, we propose DrUM, a novel method that integrates user profiling with a transformer-based adapter to enable personalized generation through condition-level modeling in the latent space. DrUM demonstrates strong performance on large-scale datasets and seamlessly integrates with open-source text encoders, making it compatible with widely used foundation T2I models without requiring additional fine-tuning.

Citations

• ViPer: Visual Personalization of Generative Models via Individual Preference Learning
• Text-to-Model: Text-Conditioned Neural Network Diffusion for Train-Once-for-All Personalization
• What matters when building vision-language models?
• PMG : Personalized Multimodal Generation with Large Language Models
• Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
• Chaining text-to-image and large language model: A novel approach for generating personalized e-commerce banners
• Diffusion Model-Based Image Editing: A Survey
• A New Creative Generation Pipeline for Click-Through Rate with Stable Diffusion Model
• Investigating Personalization Methods in Text to Music Generation
• Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization
• StyleDiffusion: Controllable Disentangled Style Transfer via Diffusion Models
• FABRIC: Personalizing Diffusion Models with Iterative Feedback
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
• Adding Conditional Control to Text-to-Image Diffusion Models
• Adding Conditional Control to Text-to-Image Diffusion Models
• InstructPix2Pix: Learning to Follow Image Editing Instructions
• DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
• DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
• Classifier-Free Diffusion Guidance
• Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
• Hierarchical Text-Conditional Image Generation with CLIP Latents
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
• Personalized Popular Music Generation Using Imitation and Structure
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Cited by

• Personalized Reward Modeling for Text-to-Image Generation

Related

• T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models
• PaRa: Personalizing Text-to-Image Diffusion via Parameter Rank Reduction
• DreamBoothDPO: Improving Personalized Generation using Direct Preference Optimization
• Hollowed Net for On-Device Personalization of Text-to-Image Diffusion Models
• Diffusion Beats Autoregressive: An Evaluation of Compositional Generation in Text-to-Image Models