Make It Count: Text-to-Image Generation with an Accurate Number of Objects

2024/06/14 by Lital Binyamin, Yoad Tewel, Binyamin, Lital +9 · 25 citations
Computer Science · Engineering · Mathematics · #Arithmetic #Artificial Intelligence (cs.AI) #Artificial intelligence #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Computer vision #FOS: Computer and information sciences #Graphics (cs.GR) #Human Motion and Animation #Image (mathematics) #Image Retrieval and Classification Techniques #Mathematics #Video Analysis and Summarization

paper · pdf · doi:10.48550/arxiv.2406.10210

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/06/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Despite the unprecedented success of text-to-image diffusion models, controlling the number of depicted objects using text is surprisingly hard. This is important for various applications from technical documents, to children's books to illustrating cooking recipes. Generating object-correct counts is fundamentally challenging because the generative model needs to keep a sense of separate identity for every instance of the object, even if several objects look identical or overlap, and then carry out a global computation implicitly during generation. It is still unknown if such representations exist. To address count-correct generation, we first identify features within the diffusion model that can carry the object identity information. We then use them to separate and count instances of objects during the denoising process and detect over-generation and under-generation. We fix the latter by training a model that predicts both the shape and location of a missing object, based on the layout of existing ones, and show how it can be used to guide denoising with correct object count. Our approach, CountGen, does not depend on external source to determine object layout, but rather uses the prior from the diffusion model itself, creating prompt-dependent and seed-dependent layouts. Evaluated on two benchmark datasets, we find that CountGen strongly outperforms the count-accuracy of existing baselines.

Cited by

• Exploring the AI Obedience: Why is Generating a Pure Color Image Harder than CyberPunk?
• D2D: Detector-to-Differentiable Critic for Improved Numeracy in Text-to-Image Generation
• DeLeaker: Dynamic Inference-Time Reweighting For Semantic Leakage Mitigation in Text-to-Image Models
• Demystifying Numerosity in Diffusion Models -- Limitations and Remedies
• Your Vision-Language Model Can't Even Count to 20: Exposing the Failures of VLMs in Compositional Counting
• OverLayBench: A Benchmark for Layout-to-Image Generation with Dense Overlaps
• CARINOX: Inference-time Scaling with Category-Aware Reward-based Initial Noise Optimization and Exploration
• Data-Driven Loss Functions for Inference-Time Optimization in Text-to-Image Generation
• CountLoop: Training-Free High-Instance Image Generation via Iterative Agent Guidance
• Too Easily Fooled? Prompt Injection Breaks LLMs on Frustratingly Simple Multiple-Choice Questions
• CountCluster: Training-Free Object Quantity Guidance with Cross-Attention Map Clustering for Text-to-Image Generation
• YOLO-Count: Differentiable Object Counting for Text-to-Image Generation
• Prompt-Free Conditional Diffusion for Multi-object Image Augmentation
• Why Settle for Mid: A Probabilistic Viewpoint to Spatial Relationship Alignment in Text-to-image Models
• Cost-Aware Routing for Efficient Text-To-Image Generation
• Lay-Your-Scene: Natural Scene Layout Generation with Diffusion Transformers
• CountDiffusion: Text-to-Image Synthesis with Training-Free Counting-Guidance Diffusion
• Generating Pedagogically Meaningful Visuals for Math Word Problems: A New Benchmark and Analysis of Text-to-Image Models
• IMAGHarmony: Controllable Image Editing with Consistent Object Quantity and Layout
• Psi-Sampler: Initial Particle Sampling for SMC-Based Inference-Time Reward Alignment in Score Models
• FlowMo: Variance-Based Flow Guidance for Coherent Motion in Video Generation
• OSPO: Object-Centric Self-Improving Preference Optimization for Text-to-Image Generation
• ISAC: Training-Free Instance-to-Semantic Attention Control for Multi-Instance Generation
• JEDI: The Force of Jensen-Shannon Divergence in Disentangling Diffusion Models
• Towards Self-Improvement of Diffusion Models via Group Preference Optimization

Related

• Counting Guidance for High Fidelity Text-to-Image Synthesis
• Detection-Driven Object Count Optimization for Text-to-Image Diffusion Models
• CountDiffusion: Text-to-Image Synthesis with Training-Free Counting-Guidance Diffusion
• Text-to-Image Diffusion Models Cannot Count, and Prompt Refinement Cannot Help
• CountCluster: Training-Free Object Quantity Guidance with Cross-Attention Map Clustering for Text-to-Image Generation