An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion
2022/08/02 by Rinon Gal, Gal, Rinon, Yuval Alaluf +11 · 249 citations
Arts and Humanities · Computer Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Digital Humanities and Scholarship #FOS: Computer and information sciences #Graphics (cs.GR) #Handwritten Text Recognition Techniques #Machine Learning (cs.LG) #Multimodal Machine Learning Applications
paper · pdf · doi:10.48550/arxiv.2208.01618
openalex publication_date 2022/08/02 · openalex created_date 2022/08/04 · openalex updated_date 2026/07/29
Abstract
Text-to-image models offer unprecedented freedom to guide creation through natural language. Yet, it is unclear how such freedom can be exercised to generate images of specific unique concepts, modify their appearance, or compose them in new roles and novel scenes. In other words, we ask: how can we use language-guided models to turn our cat into a painting, or imagine a new product based on our favorite toy? Here we present a simple approach that allows such creative freedom. Using only 3-5 images of a user-provided concept, like an object or a style, we learn to represent it through new "words" in the embedding space of a frozen text-to-image model. These "words" can be composed into natural language sentences, guiding personalized creation in an intuitive way. Notably, we find evidence that a single word embedding is sufficient for capturing unique and varied concepts. We compare our approach to a wide range of baselines, and demonstrate that it can more faithfully portray the concepts across a range of applications and tasks. Our code, data and new words will be available at: https://textual-inversion.github.io
Cited by
- Omni-Attribute: Open-vocabulary Attribute Encoder for Visual Concept Personalization
- D-OPSD: On-Policy Self-Distillation for Continuously Tuning Step-Distilled Diffusion Models
- Reverse Personalization
- M-ErasureBench: A Comprehensive Multimodal Evaluation Benchmark for Concept Erasure in Diffusion Models
- EmoCtrl: Controllable Emotional Image Content Generation
- Characterizing Motion Encoding in Video Diffusion Timesteps
- To Erase, or Not to Erase: Robust Training-Free Concept Erasure with Preservation aware Adaptive Ranked Subspace Expansion
- TreeAdapter: Hierarchical Taxonomy-Guided Adapter Composition for Fine-Grained Species Image Generation
- MorphUNet: Alpha-Controlled Biometric Transport for Diffusion-Based Face Morphing Attacks
- Extracting Voice Styles from Frozen TTS Models via Gradient-Based Inverse Optimization
- Harm is not Universal: Community-Specific Toxicity Detection is Urgently Needed
- Search Beyond What Can Be Taught: Evolving the Knowledge Boundary in Agentic Visual Generation
- FluencyVE: Marrying Temporal-Aware Mamba with Bypass Attention for Video Editing
- Bring My Cup! Personalizing Vision-Language-Action Models with Visual Attentive Prompting
- Over++: Generative Video Compositing for Layer Interaction Effects
- Finer-Personalization Rank: Fine-Grained Retrieval Examines Identity Preservation for Personalized Generation
- 3SGen: Unified Subject, Style, and Structure-Driven Image Generation with Adaptive Task-specific Memory
- DVI: Disentangling Semantic and Visual Identity for Training-Free Personalized Generation
- LumiCtrl : Learning Illuminant Prompts for Lighting Control in Personalized Text-to-Image Models
- DeContext as Defense: Safe Image Editing in Diffusion Transformers
- HiFi-Portrait: Zero-shot Identity-preserved Portrait Generation with High-fidelity Multi-face Fusion
- Elastic3D: Controllable Stereo Video Conversion with Guided Latent Decoding
- Beyond the Visible: Disocclusion-Aware Editing via Proxy Dynamic Graphs
- Directional Textual Inversion for Personalized Text-to-Image Generation
- Reducing Domain Gap with Diffusion-Based Domain Adaptation for Cell Counting
- SMRABooth: Subject and Motion Representation Alignment for Customized Video Generation
- V-Warper: Appearance-Consistent Video Diffusion Personalization via Value Warping
- AlcheMinT: Fine-grained Temporal Control for Multi-Reference Consistent Video Generation
- Targeted Data Protection for Diffusion Model by Matching Training Trajectory
- DynaIP: Dynamic Image Prompt Adapter for Scalable Zero-shot Personalized Text-to-Image Generation
- Composing Concepts from Images and Videos via Concept-prompt Binding
- Exposing Hidden Biases in Text-to-Image Models via Automated Prompt Search
- ContextDrag: Precise Drag-Based Image Editing via Context-Preserving Token Injection and Position-Aligned Attention
- OpenSubject: Leveraging Video-Derived Identity and Diversity Priors for Subject-driven Image Generation and Manipulation
- Training-free Clothing Region of Interest Self-correction for Virtual Try-On
- Beyond Hallucinations: A Multimodal-Guided Task-Aware Generative Image Compression for Ultra-Low Bitrate
- Bring Your Dreams to Life: Continual Text-to-Video Customization
- 2K-Characters-10K-Stories: A Quality-Gated Stylized Narrative Dataset with Disentangled Control and Sequence Consistency
- Few-Shot Open-Vocabulary Remote Sensing Segmentation via Textual Inversion
- Not All Birds Look The Same: Identity-Preserving Generation For Birds
- Training for Identity, Inference for Controllability: A Unified Approach to Tuning-Free Face Personalization
- Zero-Shot Video Translation and Editing with Frame Spatial-Temporal Correspondence
- GaussianBlender: Instant Stylization of 3D Gaussians with Disentangled Latent Spaces
- In-Context Sync-LoRA for Portrait Video Editing
- StyleYourSmile: Cross-Domain Face Retargeting Without Paired Multi-Style Data
- PhyCustom: Towards Realistic Physical Customization in Text-to-Image Generation
- TokenPure: Watermark Removal through Tokenized Appearance and Structural Guidance
- PSR: Scaling Multi-Subject Personalized Image Generation with Pairwise Subject-Consistency Rewards
- DreamingComics: A Story Visualization Pipeline via Subject and Layout Customized Generation using Video Models
- Controllable 3D Object Generation with Single Image Prompt
- Assimilation Matters: Model-level Backdoor Detection in Vision-Language Pretrained Models
- MultiBanana: A Challenging Benchmark for Multi-Reference Text-to-Image Generation
- Canvas-to-Image: Compositional Image Generation with Multimodal Controls
- Semantic Anchoring for Robust Personalization in Text-to-Image Diffusion Models
- AuthenLoRA: Entangling Stylization with Imperceptible Watermarks for Copyright-Secure LoRA Adapters
- MUSE: Manipulating Unified Framework for Synthesizing Emotions in Images via Test-Time Optimization
- HMARK: Radioactive Multi-Bit Semantic-Latent Watermarking for Diffusion Models
- Training-Free Diffusion Priors for Text-to-Image Generation via Optimization-based Visual Inversion
- MapReduce LoRA: Advancing the Pareto Front in Multi-Preference Optimization for Generative Models
- The Consistency Critic: Correcting Inconsistencies in Generated Images via Reference-Guided Attentive Alignment
- Delta Sampling: Data-Free Knowledge Transfer Across Diffusion Models
- A Training-Free Approach for Multi-ID Customization via Attention Adjustment and Spatial Control
- HiCoGen: Hierarchical Compositional Text-to-Image Generation in Diffusion Models via Reinforcement Learning
- Training-Free Generation of Diverse and High-Fidelity Images via Prompt Semantic Space Optimization
- Evaluating Dataset Watermarking for Fine-tuning Traceability of Customized Diffusion Models: A Comprehensive Benchmark and Removal Approach
- Rethinking Garment Conditioning in Diffusion-based Virtual Try-On
- Uni-DAD: Unified Distillation and Adaptation of Diffusion Models for Few-step Few-shot Image Generation
- FeRA: Frequency-Energy Constrained Routing for Effective Diffusion Adaptation Fine-Tuning
- MINDiff: Mask-Integrated Negative Attention for Controlling Overfitting in Text-to-Image Personalization
- SceneDesigner: Controllable Multi-Object Image Generation with 9-DoF Pose Manipulation
- PairHuman: A High-Fidelity Photographic Dataset for Customized Dual-Person Generation
- Gaussian See, Gaussian Do: Semantic 3D Motion Transfer from Multiview Video
- Which Way from B to A: The role of embedding geometry in image interpolation for Stable Diffusion
- Selecting Fine-Tuning Examples by Quizzing VLMs
- BeyondFacial: Identity-Preserving Personalized Generation Beyond Facial Close-ups
- A Style is Worth One Code: Unlocking Code-to-Style Image Generation with Discrete Style Space
- Expanding the Content-Style Frontier: a Balanced Subspace Blending Approach for Content-Style LoRA Fusion
- Harnessing Diffusion-Generated Synthetic Images for Fair Image Classification
- Perturb a Model, Not an Image: Towards Robust Privacy Protection via Anti-Personalized Diffusion Models
- Taming Identity Consistency and Prompt Diversity in Diffusion Models via Latent Concatenation and Masked Conditional Flow Matching
- A Dual-stage Prompt-driven Privacy-preserving Paradigm for Person Re-Identification
- Prompt-Based Safety Guidance Is Ineffective for Unlearned Text-to-Image Diffusion Models
- Finetuning-Free Personalization of Text to Image Generation via Hypernetworks
- NSYNC: Negative Synthetic Image Generation for Contrastive Training to Improve Stylized Text-To-Image Translation
- Deciphering Personalization: Towards Fine-Grained Explainability in Natural Language for Personalized Image Generation Models
- MIFO: Learning and Synthesizing Multi-Instance from One Image
- ID-Crafter: VLM-Grounded Online RL for Compositional Multi-Subject Video Generation
- MagicView: Multi-View Consistent Identity Customization via Priors-Guided In-Context Learning
- E-MMDiT: Revisiting Multimodal Diffusion Transformer Design for Fast Image Synthesis under Limited Resources
- BlurGuard: A Simple Approach for Robustifying Image Protection Against AI-Powered Editing
- FreeSliders: Training-Free, Modality-Agnostic Concept Sliders for Fine-Grained Diffusion Control in Images, Audio, and Video
- SplitFlow: Flow Decomposition for Inversion-Free Text-to-Image Editing
- ScalablePromptus: Scalable and High-Fidelity Prompt-Based Video Streaming
- Seeking the Unfamiliar but Memorable: Conceptual Creativity as Meta-Learning
- SpatialEdit: Benchmarking Fine-Grained Image Spatial Editing
- PSTF-AttControl: Per-Subject-Tuning-Free Personalized Image Generation with Controllable Face Attributes
- Group Relative Attention Guidance for Image Editing
- UtilGen: Utility-Centric Generative Data Augmentation with Dual-Level Task Adaptation
- Beyond Inference Intervention: Identity-Decoupled Diffusion for Face Anonymization
- Beyond Objects: Contextual Synthetic Data Generation for Fine-Grained Classification
- Neural USD: An object-centric framework for iterative editing and control
- Diffusion Adaptive Text Embedding for Text-to-Image Diffusion Models
- Track, Inpaint, Resplat: Subject-driven 3D and 4D Generation with Progressive Texture Infilling
- FreeFuse: Multi-Subject LoRA Fusion via Adaptive Token-Level Routing at Test Time
- Seeing Through the Brain: New Insights from Decoding Visual Stimuli with fMRI
- SceneDecorator: Towards Scene-Oriented Story Generation with Scene Planning and Scene Consistency
- LiteDiff
- Bridging the gap to real-world language-grounded visual concept learning
- LightsOut: Diffusion-based Outpainting for Enhanced Lens Flare Removal
- Preventing Shortcuts in Adapter Training via Providing the Shortcuts
- EchoDistill: Bidirectional Concept Distillation for One-Step Diffusion Personalization
- LayerComposer: Multi-Human Personalized Generation via Layered Canvas
- Exploring Conditions for Diffusion models in Robotic Control
- Face-MakeUpV2: Facial Consistency Learning for Controllable Text-to-Image Generation
- Protein generation with embedding learning for motif diversification
- From Competition to Synergy: Unlocking Reinforcement Learning for Subject-Driven Image Generation
- ImageGem: In-the-wild Generative Image Interaction Dataset for Generative Model Personalization
- Chimera: Compositional Image Generation using Part-based Concepting
- Personalized Image Filter: Mastering Your Photographic Style
- From Mannequin to Human: A Pose-Aware and Identity-Preserving Video Generation Framework for Lifelike Clothing Display
- Learning an Image Editing Model without Image Editing Pairs
- LoRAverse: A Submodular Framework to Retrieve Diverse Adapters for Diffusion Models
- Consistent text-to-image generation via scene de-contextualization
- Salient Concept-Aware Generative Data Augmentation
- MVCustom: Multi-View Customized Diffusion via Geometric Latent Rendering and Completion
- Prompt-based Adaptation in Large-scale Vision Models: A Survey
- EPIPTrack: Rethinking Prompt Modeling with Explicit and Implicit Prompts for Multi-Object Tracking
- Continual Personalization for Diffusion Models
- Catch-Only-One: Non-Transferable Examples for Model-Specific Authorization
- ReMix: Towards a Unified View of Consistent Character Generation and Editing
- Unpacking Hateful Memes: Presupposed Context and False Claims
- Few-shot multi-token DreamBooth with LoRa for style-consistent character generation
- Multimodal Policy Internalization for Conversational Agents
- UniMMVSR: A Unified Multi-Modal Framework for Cascaded Video Super-Resolution
- MONKEY: Masking ON KEY-Value Activation Adapter for Personalization
- CIR-CoT: Towards Interpretable Composed Image Retrieval via End-to-End Chain-of-Thought Reasoning
- Concept Retrieval -- What and How?
- StyleKeeper: Prevent Content Leakage using Negative Visual Query Guidance
- DreamOmni2: Multimodal Instruction-based Editing and Generation
- SIGMA-GEN: Structure and Identity Guided Multi-subject Assembly for Image Generation
- SpotDiff: Spotting and Disentangling Interference in Feature Space for Subject-Preserving Image Generation
- Teleportraits: Training-Free People Insertion into Any Scene
- AgeBooth: Controllable Facial Aging and Rejuvenation via Diffusion Models
- ConceptSplit: Decoupled Multi-Concept Personalization of Diffusion Models via Token-wise Adaptation and Attention Disentanglement
- REAR: Rethinking Visual Autoregressive Models via Generator-Tokenizer Consistency Regularization
- ChronoEdit: Towards Temporal Reasoning for Image Editing and World Simulation
- Latent Diffusion Unlearning: Protecting Against Unauthorized Personalization Through Trajectory Shifted Perturbations
- When and Where do Events Switch in Multi-Event Video Generation?
- Image Generation Based on Image Style Extraction
- Query-Kontext: An Unified Multimodal Model for Image Generation and Editing
- EchoGen: Generating Visual Echoes in Any Scene via Feed-Forward Subject-Driven Auto-Regressive Model
- GaussEdit: Adaptive 3D Scene Editing with Text and Image Prompts
- PatchVSR: Breaking Video Diffusion Resolution Limits with Patch-wise Video Super-Resolution
- ART-VITON: Measurement-Guided Latent Diffusion for Artifact-Free Virtual Try-On
- Dragging with Geometry: From Pixels to Geometry-Guided Image Editing
- UP2You: Fast Reconstruction of Yourself from Unconstrained Photo Collections
- ASIA: Adaptive 3D Segmentation using Few Image Annotations
- Personalized Vision via Visual In-Context Learning
- Causal-Adapter: Taming Text-to-Image Diffusion for Faithful Counterfactual Generation
- Griffin: Generative Reference and Layout Guided Image Composition
- VMDiff: Visual Mixing Diffusion for Limitless Cross-Object Synthesis
- Training-Free Synthetic Data Generation with Dual IP-Adapter Guidance
- Mind-the-Glitch: Visual Correspondence for Detecting Inconsistencies in Subject-Driven Generation
- MultiCrafter: High-Fidelity Multi-Subject Generation via Disentangled Attention and Identity-Aware Preference Alignment
- DEFT: Decompositional Efficient Fine-Tuning for Text-to-Image Models
- Memory Self-Regeneration: Uncovering Hidden Knowledge in Unlearned Models
- ReviewScore: Misinformed Peer Review Detection with Large Language Models
- Does FLUX Already Know How to Perform Physically Plausible Image Composition?
- FreeInsert: Personalized Object Insertion with Geometric and Style Control
- CusEnhancer: A Zero-Shot Scene and Controllability Enhancement Method for Photo Customization via ResInversion
- Unleashing the Potential of the Semantic Latent Space in Diffusion Models for Image Dehazing
- One-shot Embroidery Customization via Contrastive LoRA Modulation
- Prompt-Guided Dual Latent Steering for Inversion Problems
- Vision-Free Retrieval: Rethinking Multimodal Search with Textual Scene Descriptions
- ComposeMe: Attribute-Specific Image Prompts for Controllable Human Image Generation
- Stencil: Subject-Driven Generation with Context Guidance
- Animalbooth: multimodal feature enhancement for animal subject personalization
- Lynx: Towards High-Fidelity Personalized Video Generation
- AutoEdit: Automatic Hyperparameter Tuning for Image Editing
- Towards Robust Defense against Customization via Protective Perturbation Resistant to Diffusion-based Purification
- StyleProtect: Safeguarding Artistic Identity in Fine-tuned Diffusion Models
- Do It Yourself (DIY): Modifying Images for Poems in a Zero-Shot Setting Using Weighted Prompt Manipulation
- Immunizing Images from Text to Image Editing via Adversarial Cross-Attention
- Align 3D Representation and Text Embedding for 3D Content Personalization
- Delta-SVD: Efficient Compression for Personalized Text-to-Image Models
- Prompt-Driven Image Analysis with Multimodal Generative AI: Detection, Segmentation, Inpainting, and Interpretation
- Are Targeted Data Poisoning Attacks as Effective as We Think?
- UMO: Scaling Multi-Identity Consistency for Image Customization via Matching Reward
- TIDE: Achieving Balanced Subject-Driven Image Generation via Target-Instructed Diffusion Enhancement
- MV-RAG: Retrieval Augmented Multiview Diffusion
- DreamAudio: Customized Text-to-Audio Generation with Diffusion Models
- EditIDv2: Editable ID Customization with Data-Lubricated ID Feature Integration for Text-to-Image Generation
- SuMa: A Subspace Mapping Approach for Robust and Effective Concept Erasure in Text-to-Image Diffusion Models
- Towards High-Fidelity, Identity-Preserving Real-Time Makeup Transfer: Decoupling Style Generation
- Enhancing Zero-Shot Pedestrian Attribute Recognition with Synthetic Data Generation: A Comparative Study with Image-To-Image Diffusion Models
- A Data-Centric Approach to Pedestrian Attribute Recognition: Synthetic Augmentation via Prompt-driven Diffusion Models
- PromptFlare: Prompt-Generalized Defense via Cross-Attention Decoy in Diffusion-Based Inpainting
- FocusDPO: Dynamic Preference Optimization for Multi-Subject Personalized Image Generation via Adaptive Focus
- CompSlider: Compositional Slider for Disentangled Multiple-Attribute Image Generation
- The Name-Free Gap: Policy-Aware Stylistic Control in Music Generation
- NoiseCutMix: A Novel Data Augmentation Approach by Mixing Estimated Noise in Diffusion Models
- FLORA: Efficient Synthetic Data Generation for Object Detection in Low-Data Regimes via finetuning Flux LoRA
- Interact-Custom: Customized Human Object Interaction Image Generation
- VoxHammer: Training-Free Precise and Coherent 3D Editing in Native 3D Space
- Articulate3D: Zero-Shot Text-Driven 3D Object Posing
- One-shot Unsupervised Domain Adaptation with Personalized Diffusion Models
- USO: Unified Style and Subject-Driven Generation via Disentangled and Reward Learning
- DreamSwapV: Mask-guided Subject Swapping for Any Customized Video Editing
- MUSE: Multi-Subject Unified Synthesis via Explicit Layout Semantic Expansion
- CTA-Flux: Integrating Chinese Cultural Semantics into High-Quality English Text-to-Image Communities
- OmniTry: Virtual Try-On Anything without Masks
- Single-Reference Text-to-Image Manipulation with Dual Contrastive Denoising Score
- Express4D: Expressive, Friendly, and Extensible 4D Facial Motion Generation Benchmark
- SPG: Style-Prompting Guidance for Style-Specific Content Creation
- MM-R1: Unleashing the Power of Unified Multimodal Large Language Models for Personalized Image Generation
- Noise Matters: Optimizing Matching Noise for Diffusion Classifiers
- A Segmentation-driven Editing Method for Bolt Defect Augmentation and Detection
- Increasing the Utility of Synthetic Images through Chamfer Guidance
- Enhancing Diffusion Face Generation with Contrastive Embeddings and SegFormer Guidance
- Gen-AFFECT: Generation of Avatar Fine-grained Facial Expressions with Consistent identiTy
- Per-Query Visual Concept Learning
- Lay2Story: Extending Diffusion Transformers for Layout-Togglable Story Generation
- TARA: Token-Aware LoRA for Composable Personalization in Diffusion Models
- CObL: Toward Zero-Shot Ordinal Layering without User Prompting
- Cut2Next: Generating Next Shot via In-Context Tuning
- Learning User Preferences for Image Generation Model
- X2Edit: Revisiting Arbitrary-Instruction Image Editing through Self-Constructed Data and Task-Aware Representation Learning
- MAViS: A Multi-Agent Framework for Long-Sequence Video Storytelling
- Stand-In: A Lightweight and Plug-and-Play Identity Control for Video Generation
- Comparison Reveals Commonality: Customized Image Generation through Contrastive Inversion
- Enhancing Small-Scale Dataset Expansion with Triplet-Connection-based Sample Re-Weighting
- SketchAnimator: Animate Sketch via Motion Customization of Text-to-Video Diffusion Models
- DCoAR: Deep Concept Injection into Unified Autoregressive Models for Personalized Text-to-Image Generation
- NEP: Autoregressive Image Editing via Next Editing Token Prediction
- Textual Inversion for Efficient Adaptation of Open-Vocabulary Object Detectors Without Forgetting
- Voost: A Unified and Scalable Diffusion Transformer for Bidirectional Virtual Try-On and Try-Off
- Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting
- Edge-Assisted Collaborative Fine-Tuning for Multi-User Personalized Artificial Intelligence Generated Content (AIGC)
- SCFlow: Implicitly Learning Style and Content Disentanglement with Flow Models
- Zero Shot Domain Adaptive Semantic Segmentation by Synthetic Data Generation and Progressive Adaptation
- Subject or Style: Adaptive and Training-Free Mixture of LoRAs
- PromptSafe: Gated Prompt Tuning for Safe Text-to-Image Generation
- YOLO-Count: Differentiable Object Counting for Text-to-Image Generation
- Steering Guidance for Personalized Text-to-Image Diffusion Models
- StorySync: Training-Free Subject Consistency in Text-to-Image Generation via Region Harmonization
- UniEmo: Unifying Emotional Understanding and Generation with Learnable Expert Queries
- Hate in Plain Sight: On the Risks of Moderating AI-Generated Hateful Illusions
- Aether Weaver: Multimodal Affective Narrative Co-Generation with Dynamic Scene Graphs
- Adapting Vehicle Detectors for Aerial Imagery to Unseen Domains with Weak Supervision
Related