DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
2023/06/01 by Nataniel Ruiz, Yuanzhen Li, Varun Jampani +3 · 215 citations
Computer Science · #Computer Graphics and Visualization Techniques #Generative Adversarial Networks and Image Synthesis #Video Analysis and Summarization
paper · doi:10.1109/cvpr52729.2023.02155
openalex publication_date 2023/06/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/29
Abstract
Large text-to-image models achieved a remarkable leap in the evolution of AI, enabling high-quality and diverse synthesis of images from a given text prompt. However, these models lack the ability to mimic the appearance of subjects in a given reference set and synthesize novel renditions of them in different contexts. In this work, we present a new approach for “personalization” of text-to-image diffusion models. Given as input just a few images of a subject, we fine-tune a pretrained text-to-image model such that it learns to bind a unique identifier with that specific subject. Once the subject is embedded in the output domain of the model, the unique identifier can be used to synthesize novel photorealistic images of the subject contextualized in different scenes. By leveraging the semantic prior embedded in the model with a new autogenous class-specific prior preservation loss, our technique enables synthesizing the subject in diverse scenes, poses, views and lighting conditions that do not appear in the reference images. We apply our technique to several previously-unassailable tasks, including subject recontextualization, text-guided view synthesis, and artistic rendering, all while preserving the subject's key features. We also provide a new dataset and evaluation protocol for this new task of subject-driven generation. Project page: https://dreambooth.github.io/
Citations
Cited by
- Survey of Cultural Awareness in Language Models: Text and Beyond
- Domain Generalization for Semantic Segmentation: A Survey
- A Survey of Multimodal Controllable Diffusion Models
- SpatialEdit: Benchmarking Fine-Grained Image Spatial Editing
- Noise Aggregation Analysis Driven by Small-Noise Injection: Efficient Membership Inference for Diffusion Models
- Diffusion-Driven Progressive Target Manipulation for Source-Free Domain Adaptation
- PSTF-AttControl: Per-Subject-Tuning-Free Personalized Image Generation with Controllable Face Attributes
- Group Relative Attention Guidance for Image Editing
- DiffusionX: Efficient Edge-Cloud Collaborative Image Generation with Multi-Round Prompt Evolution
- Rethinking Visual Intelligence: Insights from Video Pretraining
- TokenAR: Multiple Subject Generation via Autoregressive Token-level enhancement
- Kernelized Sparse Fine-Tuning with Bi-level Parameter Competition for Vision Models
- Beyond Inference Intervention: Identity-Decoupled Diffusion for Face Anonymization
- Beyond Objects: Contextual Synthetic Data Generation for Fine-Grained Classification
- Neural USD: An object-centric framework for iterative editing and control
- State of the Art on Diffusion Models for Visual Computing
- Rethinking Noise Sampling in Class-Imbalanced Diffusion Models
- Track, Inpaint, Resplat: Subject-driven 3D and 4D Generation with Progressive Texture Infilling
- FreeFuse: Multi-Subject LoRA Fusion via Adaptive Token-Level Routing at Test Time
- Seeing Through the Brain: New Insights from Decoding Visual Stimuli with fMRI
- LiteDiff
- Enhancing Video Inpainting with Aligned Frame Interval Guidance
- LightsOut: Diffusion-based Outpainting for Enhanced Lens Flare Removal
- Preventing Shortcuts in Adapter Training via Providing the Shortcuts
- EchoDistill: Bidirectional Concept Distillation for One-Step Diffusion Personalization
- AccuQuant: Simulating Multiple Denoising Steps for Quantizing Diffusion Models
- EditInfinity: Image Editing with Binary-Quantized Generative Models
- LayerComposer: Multi-Human Personalized Generation via Layered Canvas
- GenColorBench: A Color Evaluation Benchmark for Text-to-Image Generation Models
- Face-MakeUpV2: Facial Consistency Learning for Controllable Text-to-Image Generation
- Exposing Blindspots: Cultural Bias Evaluation in Generative Image Models
- From Competition to Synergy: Unlocking Reinforcement Learning for Subject-Driven Image Generation
- ImageGem: In-the-wild Generative Image Interaction Dataset for Generative Model Personalization
- Kaleido: Open-Sourced Multi-Subject Reference Video Generation Model
- Beyond Real Faces: Synthetic Datasets Can Achieve Reliable Recognition Performance without Privacy Compromise
- Chimera: Compositional Image Generation using Part-based Concepting
- Personalized Image Filter: Mastering Your Photographic Style
- The Face of Persuasion: Analyzing Bias and Generating Culture-Aware Ads
- Learning an Image Editing Model without Image Editing Pairs
- WithAnyone: Towards Controllable and ID Consistent Image Generation
- LoRAverse: A Submodular Framework to Retrieve Diverse Adapters for Diffusion Models
- Consistent text-to-image generation via scene de-contextualization
- Salient Concept-Aware Generative Data Augmentation
- MVCustom: Multi-View Customized Diffusion via Geometric Latent Rendering and Completion
- SceneAdapt: Scene-aware Adaptation of Human Motion Diffusion
- FedMMKT:Co-Enhancing a Server Text-to-Image Model and Client Task Models in Multi-Modal Federated Learning
- Point Prompting: Counterfactual Tracking with Video Diffusion Models
- Continual Personalization for Diffusion Models
- DragFlow: Unleashing DiT Priors with Region Based Supervision for Drag Editing
- ReMix: Towards a Unified View of Consistent Character Generation and Editing
- CharCom: Composable Identity Control for Multi-Character Story Illustration
- Cross-Sensor Touch Generation
- Few-shot multi-token DreamBooth with LoRa for style-consistent character generation
- Multimodal Policy Internalization for Conversational Agents
- InstructX: Towards Unified Visual Editing with MLLM Guidance
- MONKEY: Masking ON KEY-Value Activation Adapter for Personalization
- StyleKeeper: Prevent Content Leakage using Negative Visual Query Guidance
- DreamOmni2: Multimodal Instruction-based Editing and Generation
- SIGMA-GEN: Structure and Identity Guided Multi-subject Assembly for Image Generation
- SpotDiff: Spotting and Disentangling Interference in Feature Space for Subject-Preserving Image Generation
- Teleportraits: Training-Free People Insertion into Any Scene
- AgeBooth: Controllable Facial Aging and Rejuvenation via Diffusion Models
- SAEdit: Token-level control for continuous image editing via Sparse AutoEncoder
- ConceptSplit: Decoupled Multi-Concept Personalization of Diffusion Models via Token-wise Adaptation and Attention Disentanglement
- REAR: Rethinking Visual Autoregressive Models via Generator-Tokenizer Consistency Regularization
- Latent Diffusion Unlearning: Protecting Against Unauthorized Personalization Through Trajectory Shifted Perturbations
- When and Where do Events Switch in Multi-Event Video Generation?
- Image Generation Based on Image Style Extraction
- IMAGEdit: Let Any Subject Transform
- MOLM: Mixture of LoRA Markers
- Stitch: Training-Free Position Control in Multimodal Diffusion Transformers
- Query-Kontext: An Unified Multimodal Model for Image Generation and Editing
- dParallel: Learnable Parallel Decoding for dLLMs
- EchoGen: Generating Visual Echoes in Any Scene via Feed-Forward Subject-Driven Auto-Regressive Model
- GaussEdit: Adaptive 3D Scene Editing with Text and Image Prompts
- ART-VITON: Measurement-Guided Latent Diffusion for Artifact-Free Virtual Try-On
- Dragging with Geometry: From Pixels to Geometry-Guided Image Editing
- OmniDFA: A Unified Framework for Open Set Synthesis Image Detection and Few-Shot Attribution
- Environment-Aware Satellite Image Generation with Diffusion Models
- UP2You: Fast Reconstruction of Yourself from Unconstrained Photo Collections
- TR2-D2: Tree Search Guided Trajectory-Aware Fine-Tuning for Discrete Diffusion
- MapGenerator: a framework for learning a diffusion model for text promptable map generation
- Personalized Vision via Visual In-Context Learning
- SIE3D: Single-image Expressive 3D Avatar generation via Semantic Embedding and Perceptual Expression Loss
- ReLumix: Extending Image Relighting to Video via Video Diffusion Models
- Griffin: Generative Reference and Layout Guided Image Composition
- Rethinking Inter-LoRA Orthogonality in Adapter Merging: Insights from Orthogonal Monte Carlo Dropout
- MMPB: It's Time for Multi-Modal Personalization
- Training-Free Synthetic Data Generation with Dual IP-Adapter Guidance
- High-Quality Sound Separation Across Diverse Categories via Visually-Guided Generative Modeling
- Mind-the-Glitch: Visual Correspondence for Detecting Inconsistencies in Subject-Driven Generation
- MultiCrafter: High-Fidelity Multi-Subject Generation via Disentangled Attention and Identity-Aware Preference Alignment
- Abductive Logical Rule Induction by Bridging Inductive Logic Programming and Multimodal Large Language Models
- DEFT: Decompositional Efficient Fine-Tuning for Text-to-Image Models
- Does FLUX Already Know How to Perform Physically Plausible Image Composition?
- Evaluating the Evaluators: Metrics for Compositional Text-to-Image Generation
- Task-Oriented Computation Offloading for Edge Inference: An Integrated Bayesian Optimization and Deep Reinforcement Learning Framework
- FreeInsert: Personalized Object Insertion with Geometric and Style Control
- CusEnhancer: A Zero-Shot Scene and Controllability Enhancement Method for Photo Customization via ResInversion
- Efficient Encoder-Free Pose Conditioning and Pose Control for Virtual Try-On
- Unleashing the Potential of the Semantic Latent Space in Diffusion Models for Image Dehazing
- Latent Iterative Refinement Flow: A Geometric-Constrained Approach for Few-Shot Generation
- From Synthetic to Real: Toward Identity-Consistent Makeup Transfer with Synthetic and Real Data
- Hand-Object Interaction in the Age of Large Foundation Models:Reconstruction, Generation, and Embodied Transfer
- Instant Preference Alignment for Text-to-Image Diffusion Models
- Prompt-Guided Dual Latent Steering for Inversion Problems
- Seg4Diff: Unveiling Open-Vocabulary Segmentation in Text-to-Image Diffusion Transformers
- ContextFlow: Training-Free Video Object Editing via Adaptive Context Enrichment
- CARINOX: Inference-time Scaling with Category-Aware Reward-based Initial Noise Optimization and Exploration
- Stencil: Subject-Driven Generation with Context Guidance
- Animalbooth: multimodal feature enhancement for animal subject personalization
- Follow-Your-Emoji-Faster: Towards Efficient, Fine-Controllable, and Expressive Freestyle Portrait Animation
- Lynx: Towards High-Fidelity Personalized Video Generation
- AutoEdit: Automatic Hyperparameter Tuning for Image Editing
- Towards Robust Defense against Customization via Protective Perturbation Resistant to Diffusion-based Purification
- Condition Weaving Meets Expert Modulation: Towards Universal and Controllable Image Generation
- StyleProtect: Safeguarding Artistic Identity in Fine-tuned Diffusion Models
- PosBridge: Multi-View Positional Embedding Transplant for Identity-Aware Image Editing
- End4: End-to-end Denoising Diffusion for Diffusion-Based Inpainting Detection
- HoloGarment: 360° Novel View Synthesis of In-the-Wild Garments
- Styleclone: Face Stylization with Diffusion Based Data Augmentation
- Immunizing Images from Text to Image Editing via Adversarial Cross-Attention
- Delta-SVD: Efficient Compression for Personalized Text-to-Image Models
- Prompt-Driven Image Analysis with Multimodal Generative AI: Detection, Segmentation, Inpainting, and Interpretation
- UMO: Scaling Multi-Identity Consistency for Image Customization via Matching Reward
- TIDE: Achieving Balanced Subject-Driven Image Generation via Target-Instructed Diffusion Enhancement
- MV-RAG: Retrieval Augmented Multiview Diffusion
- DreamAudio: Customized Text-to-Audio Generation with Diffusion Models
- EditIDv2: Editable ID Customization with Data-Lubricated ID Feature Integration for Text-to-Image Generation
- Human Preference-Aligned Concept Customization Benchmark via Decomposed Evaluation
- Towards High-Fidelity, Identity-Preserving Real-Time Makeup Transfer: Decoupling Style Generation
- Data-Driven Loss Functions for Inference-Time Optimization in Text-to-Image Generation
- EM3M: An Electron Micrograph Dataset for Microstructural Segmentation and Generation
- MOSAIC: Multi-Subject Personalized Generation via Correspondence-Aware Alignment and Disentanglement
- Discrete Noise Inversion for Next-scale Autoregressive Text-based Image Editing
- FocusDPO: Dynamic Preference Optimization for Multi-Subject Personalized Image Generation via Adaptive Focus
- CompSlider: Compositional Slider for Disentangled Multiple-Attribute Image Generation
- DevilSight: Augmenting Monocular Human Avatar Reconstruction through a Virtual Perspective
- FLORA: Efficient Synthetic Data Generation for Object Detection in Low-Data Regimes via finetuning Flux LoRA
- Efficient Diffusion Models: A Comprehensive Survey From Principles to Practices
- Inference-Time Alignment Control for Diffusion Models with Reinforcement Learning Guidance
- FastFit: Accelerating Multi-Reference Virtual Try-On via Cacheable Diffusion Models
- IELDG: Suppressing Domain-Specific Noise with Inverse Evolution Layers for Domain Generalized Semantic Segmentation
- Interact-Custom: Customized Human Object Interaction Image Generation
- Bi-LoRA: Efficient Sharpness-Aware Minimization for Fine-Tuning Large-Scale Models
- Articulate3D: Zero-Shot Text-Driven 3D Object Posing
- One-shot Unsupervised Domain Adaptation with Personalized Diffusion Models
- USO: Unified Style and Subject-Driven Generation via Disentangled and Reward Learning
- Generative AI in Map-Making: A Technical Exploration and Its Implications for Cartographers
- VQualA 2025 Challenge on Face Image Quality Assessment: Methods and Results
- GenTune: Toward Traceable Prompts to Improve Controllability of Image Refinement in Environment Design
- Scaling Group Inference for Diverse and High-Quality Generation
- DreamSwapV: Mask-guided Subject Swapping for Any Customized Video Editing
- MUSE: Multi-Subject Unified Synthesis via Explicit Layout Semantic Expansion
- Single-Reference Text-to-Image Manipulation with Dual Contrastive Denoising Score
- Leveraging Diffusion Models for Stylization using Multiple Style Images
- SPG: Style-Prompting Guidance for Style-Specific Content Creation
- LoRAtorio: An intrinsic approach to LoRA Skill Composition
- MM-R1: Unleashing the Power of Unified Multimodal Large Language Models for Personalized Image Generation
- Match & Choose: Model Selection Framework for Fine-tuning Text-to-Image Diffusion Models
- A Segmentation-driven Editing Method for Bolt Defect Augmentation and Detection
- Increasing the Utility of Synthetic Images through Chamfer Guidance
- TweezeEdit: Consistent and Efficient Image Editing with Path Regularization
- Translation of Text Embedding via Delta Vector to Suppress Strongly Entangled Content in Text-to-Image Diffusion Models
- Efficient Forward-Only Data Valuation for Pretrained LLMs and VLMs
- SynSpill: Improved Industrial Spill Detection With Synthetic Data
- Improving watermelon (Citrullus lanatus) disease classification with generative artificial intelligence (GenAI)-based synthetic and real-field images via a custom EfficientNetV2-L model
- A Survey on 3D Gaussian Splatting Applications: Segmentation, Editing, and Generation
- Dual Recursive Feedback on Generation and Appearance Latents for Pose-Robust Text-to-Image Diffusion
- Gen-AFFECT: Generation of Avatar Fine-grained Facial Expressions with Consistent identiTy
- RefAdGen: High-Fidelity Advertising Image Generation
- Per-Query Visual Concept Learning
- Lay2Story: Extending Diffusion Transformers for Layout-Togglable Story Generation
- TARA: Token-Aware LoRA for Composable Personalization in Diffusion Models
- Cut2Next: Generating Next Shot via In-Context Tuning
- Learning User Preferences for Image Generation Model
- X2Edit: Revisiting Arbitrary-Instruction Image Editing through Self-Constructed Data and Task-Aware Representation Learning
- Make Your MoVe: Make Your 3D Contents by Adapting Multi-View Diffusion Models to External Editing
- ShoulderShot: Generating Over-the-Shoulder Dialogue Videos
- MAViS: A Multi-Agent Framework for Long-Sequence Video Storytelling
- Stand-In: A Lightweight and Plug-and-Play Identity Control for Video Generation
- Comparison Reveals Commonality: Customized Image Generation through Contrastive Inversion
- SketchAnimator: Animate Sketch via Motion Customization of Text-to-Video Diffusion Models
- DCoAR: Deep Concept Injection into Unified Autoregressive Models for Personalized Text-to-Image Generation
- MultiRef: Controllable Image Generation with Multiple Visual References
- Improving Diagnostic Accuracy for Oral Cancer with inpainting Synthesis Lesions Generated Using Diffusion Models
- NEP: Autoregressive Image Editing via Next Editing Token Prediction
- Textual Inversion for Efficient Adaptation of Open-Vocabulary Object Detectors Without Forgetting
- PoseGen: In-Context LoRA Finetuning for Pose-Controllable Long Human Video Generation
- FLUX-Makeup: High-Fidelity, Identity-Consistent, and Robust Makeup Transfer via Diffusion Transformer
- Voost: A Unified and Scalable Diffusion Transformer for Bidirectional Virtual Try-On and Try-Off
- Zero-Residual Concept Erasure via Progressive Alignment in Text-to-Image Model
- Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting
- Edge-Assisted Collaborative Fine-Tuning for Multi-User Personalized Artificial Intelligence Generated Content (AIGC)
- DogFit: Domain-guided Fine-tuning for Efficient Transfer Learning of Diffusion Models
- LumiGen: An LVLM-Enhanced Iterative Framework for Fine-Grained Text-to-Image Generation
- Anti-Tamper Protection for Unauthorized Individual Image Generation
- DiWA: Diffusion Policy Adaptation with World Models
- CoEmoGen: Towards Semantically-Coherent and Scalable Emotional Image Content Generation
- Draw Your Mind: Personalized Generation via Condition-Level Modeling in Text-to-Image Diffusion Models
- VideoGuard: Protecting Video Content from Unauthorized Editing
- Zero Shot Domain Adaptive Semantic Segmentation by Synthetic Data Generation and Progressive Adaptation
- Diffusion Models with Adaptive Negative Sampling Without External Resources
- Subject or Style: Adaptive and Training-Free Mixture of LoRAs
- AttriCtrl: Fine-Grained Control of Aesthetic Attribute Intensity in Diffusion Models
- Devil is in the Detail: Towards Injecting Fine Details of Image Prompt in Image Generation via Conflict-free Guidance and Stratified Attention
- Beyond Vulnerabilities: A Survey of Adversarial Attacks as Both Threats and Defenses in Computer Vision Systems
- Practical, Generalizable and Robust Backdoor Attacks on Text-to-Image Diffusion Models
- StyleSentinel: Reliable Artistic Copyright Verification via Stylistic Fingerprints
- Steering Guidance for Personalized Text-to-Image Diffusion Models
- LAMIC: Layout-Aware Multi-Image Composition via Scalability of Multimodal Diffusion Transformer
- Adaptively Distilled ControlNet: Accelerated Training and Superior Sampling for Medical Image Synthesis
- Stable-Sim2Real: Exploring Simulation of Real-Captured 3D Data with Two-Stage Depth Diffusion
- StorySync: Training-Free Subject Consistency in Text-to-Image Generation via Region Harmonization
- UniEmo: Unifying Emotional Understanding and Generation with Learnable Expert Queries
- X-NeMo: Expressive Neural Motion Reenactment via Disentangled Latent Attention
- Hate in Plain Sight: On the Risks of Moderating AI-Generated Hateful Illusions
Related