Dynamic VLM-Guided Negative Prompting for Diffusion Models

2025/10/30 by Chang, Hoyeon, Kim, Seungjin, Choi, Yoonseok
#Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2510.26052

Abstract

We propose a novel approach for dynamic negative prompting in diffusion models that leverages Vision-Language Models (VLMs) to adaptively generate negative prompts during the denoising process. Unlike traditional Negative Prompting methods that use fixed negative prompts, our method generates intermediate image predictions at specific denoising steps and queries a VLM to produce contextually appropriate negative prompts. We evaluate our approach on various benchmark datasets and demonstrate the trade-offs between negative guidance strength and text-image alignment.

Citations

• Qwen2.5-VL Technical Report
• Dynamic Negative Guidance of Diffusion Models
• SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generation
• Understanding the Impact of Negative Prompts: When and How Do They Take Effect?
• To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Now
• Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models?
• Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Prompts
• Visual Instruction Tuning
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models
• Classifier-Free Diffusion Guidance
• BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
• High-Resolution Image Synthesis with Latent Diffusion Models
• Learning Transferable Visual Models From Natural Language Supervision
• Denoising Diffusion Probabilistic Models
• Microsoft COCO: Common Objects in Context
• Contrastive CFG: Improving CFG in Diffusion Models by Contrasting Positive and Negative Concepts

Related

• Dynamic Negative Guidance of Diffusion Models
• Diffusion Models with Adaptive Negative Sampling Without External Resources
• Advanced Image Generation: Negative Prompt Optimization and Latent Classifier Guidance
• Supercharged One-step Text-to-Image Diffusion Models with Negative Prompts
• Improving image synthesis with diffusion-negative sampling