2025/05/27 by Sanghyun Jo, Wooyeol Lee, Jo, Sanghyun +7
Computer Science · #Advanced Neural Network Applications #Benchmark (surveying) #Carving #Computer Vision and Pattern Recognition (cs.CV) #Container (type theory) #FOS: Computer and information sciences #Granularity #Key (lock) #Parallel Computing and Optimization Techniques #Reinforcement Learning in Robotics #Rotation formalisms in three dimensions #Semantics (computer science) #TRACE (psycholinguistics)
paper · pdf · doi:10.48550/arxiv.2505.20935
published in ArXiv.org
openalex publication_date 2025/05/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
Recent open-weight text-to-image (T2I) diffusion models still struggle with multi-instance prompts, often omitting or merging instances and mixing semantics among similar objects. We trace these failures to early denoising steps, before instance boundaries are reliably stabilized. Existing training-free guidance is largely driven by cross-attention or other token-conditioned semantic signals. Such guidance can separate concepts at the token level, but largely assumes that distinct instance regions have already emerged. In early denoising steps, it cannot reliably carve out these regions, so count failures and semantic mixing persist. By contrast, self-attention exposes class-agnostic instance layouts during early denoising. To exploit this asymmetry, we propose ISAC (Instance-to-Semantic Attention Control), a training-free, model-agnostic objective that first stabilizes self-attention layouts and then binds cross-attention semantics within them, without fine-tuning or external vision models. Across T2I-CompBench, HRS-Bench, and our newly curated IntraCompBench, ISAC consistently outperforms prior training-free methods. Furthermore, ISAC enhances layout-to-image controllers by refining coarse, overlapping bounding boxes into dense instance masks. Code and IntraCompBench are available at https://shjo-april.github.io/ISAC.