Plug-and-Play Prompt Refinement via Latent Feedback for Diffusion Model Alignment

2025/10/01 by Lee, Suhyeon, Ye, Jong Chul
#Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2510.00430

Abstract

Despite the recent progress, reinforcement learning (RL)-based fine-tuning of diffusion models often struggles with generalization, composability, and robustness against reward hacking. Recent studies have explored prompt refinement as a modular alternative, but most adopt a feed-forward approach that applies a single refined prompt throughout the entire sampling trajectory, thereby failing to fully leverage the sequential nature of reinforcement learning. To address this, here we introduce PromptLoop, a plug-and-play RL framework that incorporates latent feedback into step-wise prompt refinement. Rather than modifying diffusion model weights, a multimodal large language model (MLLM) is trained with RL to iteratively update prompts based on intermediate latent states of diffusion models. This design achieves a structural analogy to the Diffusion RL approach, while retaining the flexibility and generality of prompt-based alignment. Extensive experiments across diverse reward functions and diffusion backbones demonstrate that PromptLoop (i) achieves effective reward optimization, (ii) generalizes seamlessly to unseen models, (iii) composes orthogonally with existing alignment methods, and (iv) mitigates over-optimization and reward hacking.

Citations

• Dynamic Classifier-Free Diffusion Guidance via Online Feedback
• Interleaving Reasoning for Better Text-to-Image Generation
• PromptEnhancer: A Simple Approach to Enhance Text-to-Image Models via Chain-of-Thought Prompt Rewriting
• Visual-CoG: Stage-Aware Reinforcement Learning with Chain of Guidance for Text-to-Image Generation
• InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
• Reward-Agnostic Prompt Optimization for Text-to-Image Diffusion Models
• RePrompt: Reasoning-Augmented Reprompting for Text-to-Image Generation via Reinforcement Learning
• Understanding Prompt Tuning and In-Context Learning via Meta-Learning
• DanceGRPO: Unleashing GRPO on Visual Generation
• Qwen2.5-VL Technical Report
• A General Framework for Inference-time Scaling and Steering of Diffusion Models
• Test-time Alignment of Diffusion Models without Reward Over-optimization
• Free²Guide: Training-Free Text-to-Video Alignment using Image LVLM
• Golden Noise for Diffusion Models: A Learning Framework
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control
• The Llama 3 Herd of Models
• Analysis of Classifier-Free Guidance Weight Schedulers
• Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models
• Improving Text-to-Image Consistency via Automatic Prompt Optimization
• ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment
• Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
• Adversarial Diffusion Distillation
• Diffusion Model Alignment Using Direct Preference Optimization
• GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment
• SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
• Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Training Diffusion Models with Reinforcement Learning
• Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation
• Visual Instruction Tuning
• FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model
• Optimizing Prompts for Text-to-Image Generation
• Diffusion Posterior Sampling for General Noisy Inverse Problems
• Improving Diffusion Models for Inverse Problems using Manifold Constraints
• DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps
• Elucidating the Design Space of Diffusion-Based Generative Models
• Pseudo Numerical Methods for Diffusion Models on Manifolds
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• An Explanation of In-context Learning as Implicit Bayesian Inference
• The Power of Scale for Parameter-Efficient Prompt Tuning
• Learning Transferable Visual Models From Natural Language Supervision
• Score-Based Generative Modeling through Stochastic Differential Equations
• Denoising Diffusion Implicit Models
• Denoising Diffusion Probabilistic Models
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• Language Models are Few-Shot Learners
• Generative Modeling by Estimating Gradients of the Data Distribution
• BERTScore: Evaluating Text Generation with BERT
• Proximal Policy Optimization Algorithms
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• A Connection Between Score Matching and Denoising Autoencoders
• Reinforcement Learning: An Introduction

Related

• Inference-Time Alignment Control for Diffusion Models with Reinforcement Learning Guidance
• Prompt Tuning with Diffusion for Few-Shot Pre-trained Policy Generalization
• Avoiding mode collapse in diffusion models fine-tuned with reinforcement learning
• Diffusion Blend: Inference-Time Multi-Preference Alignment for Diffusion Models
• Distribution Matching Distillation Meets Reinforcement Learning