2024/04/23 by Hongyu Chen, Chen, Hongyu, Yiqi Gao +11 · 1 citation
Engineering · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Piezoelectric Actuators and Control
paper · pdf · doi:10.48550/arxiv.2404.14768
openalex publication_date 2024/04/23 · openalex created_date 2024/04/26 · openalex updated_date 2026/07/28
Recently, integrating visual controls into text-to-image~(T2I) models, such as ControlNet method, has received significant attention for finer control capabilities. While various training-free methods make efforts to enhance prompt following in T2I models, the issue with visual control is still rarely studied, especially in the scenario that visual controls are misaligned with text prompts. In this paper, we address the challenge of ``Prompt Following With Visual Control" and propose a training-free approach named Mask-guided Prompt Following (MGPF). Object masks are introduced to distinct aligned and misaligned parts of visual controls and prompts. Meanwhile, a network, dubbed as Masked ControlNet, is designed to utilize these object masks for object generation in the misaligned visual control region. Further, to improve attribute matching, a simple yet efficient loss is designed to align the attention maps of attributes with object regions constrained by ControlNet and object masks. The efficacy and superiority of MGPF are validated through comprehensive quantitative and qualitative experiments.