LLM-Enabled Style and Content Regularization for Personalized Text-to-Image Generation

2025/04/19 by Albert S. Yu, Wei Feng, Yu, Anran +10
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Image Retrieval and Classification Techniques #Topic Modeling #Video Analysis and Summarization

paper · pdf · doi:10.48550/arxiv.2504.15309

openalex publication_date 2025/04/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The personalized text-to-image generation has rapidly advanced with the emergence of Stable Diffusion. Existing methods, which typically fine-tune models using embedded identifiers, often struggle with insufficient stylization and inaccurate image content due to reduced textual controllability. In this paper, we propose style refinement and content preservation strategies. The style refinement strategy leverages the semantic information of visual reasoning prompts and reference images to optimize style embeddings, allowing a more precise and consistent representation of style information. The content preservation strategy addresses the content bias problem by preserving the model's generalization capabilities, ensuring enhanced textual controllability without compromising stylization. Experimental results verify that our approach achieves superior performance in generating consistent and personalized text-to-image outputs.

Citations

• StyleTokenizer: Defining Image Style by a Single Instance for Controlling Diffusion Models
• Unifying Visual and Semantic Feature Spaces with Diffusion Models for Enhanced Cross-Modal Alignment
• MuseumMaker: Continual Style Customization without Catastrophic Forgetting
• Selectively Informative Description can Reduce Undesired Embedding Entanglements in Text-to-Image Personalization
• Fast Personalized Text-to-Image Syntheses With Attention Injection
• Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
• Plug-in Diffusion Model for Sequential Recommendation
• ControlStyle: Text-Driven Stylized Image Generation Using Diffusion Priors
• Dense Text-to-Image Generation with Attention Modulation
• ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation
• InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning
• Anti-DreamBooth: Protecting users from personalized text-to-image synthesis
• Scaling up GANs for Text-to-Image Synthesis
• Region-Aware Diffusion for Zero-shot Text-driven Image Editing
• Optimizing Prompts for Text-to-Image Generation
• Multi-Concept Customization of Text-to-Image Diffusion
• DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
• DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
• An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion
• Exploring CLIP for Assessing the Look and Feel of Images
• Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
• Hierarchical Text-Conditional Image Generation with CLIP Latents
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• StyTr²: Image Style Transfer with Transformers
• A Neural Algorithm of Artistic Style
• CreativeSynth: Cross-Art-Attention for Artistic Image Synthesis with Multimodal Diffusion

Related

• Visual Style Prompting with Swapping Self-Attention
• Style Aligned Image Generation via Shared Attention
• ConsiStyle: Style Diversity in Training-Free Consistent T2I Generation
• StyleInject: Parameter Efficient Tuning of Text-to-Image Diffusion Models
• InstantStyle: Free Lunch towards Style-Preserving in Text-to-Image Generation