GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment
2023/10/17 by Dhruba Ghosh, Ghosh, Dhruba, Hanna Hajishirzi +3 · 241 citations
Computer Science · #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Handwritten Text Recognition Techniques
paper · pdf · doi:10.48550/arxiv.2310.11513
Abstract
Recent breakthroughs in diffusion models, multimodal pretraining, and efficient finetuning have led to an explosion of text-to-image generative models. Given human evaluation is expensive and difficult to scale, automated methods are critical for evaluating the increasingly large number of new models. However, most current automated evaluation metrics like FID or CLIPScore only offer a holistic measure of image quality or image-text alignment, and are unsuited for fine-grained or instance-level analysis. In this paper, we introduce GenEval, an object-focused framework to evaluate compositional image properties such as object co-occurrence, position, count, and color. We show that current object detection models can be leveraged to evaluate text-to-image models on a variety of generation tasks with strong human agreement, and that other discriminative vision models can be linked to this pipeline to further verify properties like object color. We then evaluate several open-source text-to-image models and analyze their relative generative capabilities on our benchmark. We find that recent models demonstrate significant improvement on these tasks, though they are still lacking in complex capabilities such as spatial relations and attribute binding. Finally, we demonstrate how GenEval might be used to help discover existing failure modes, in order to inform development of the next generation of text-to-image models. Our code to run the GenEval framework is publicly available at https://github.com/djghosh13/geneval.
Cited by
- Unison: A Fully Automatic, Task-Universal, and Low-Cost Framework for Unified Understanding and Generation
- LongCat-Image Technical Report
- ThinkGen: Generalized Thinking for Visual Generation
- D-OPSD: On-Policy Self-Distillation for Continuously Tuning Step-Distilled Diffusion Models
- Direct Diffusion Score Preference Optimization via Stepwise Contrastive Policy-Pair Supervision
- Guided Path Sampling: Steering Diffusion Models Back on Track with Principled Path Guidance
- IB-Flow: Information Bottleneck-Guided CFG Distillation for Few-Step Text-to-Image Generation
- Self-Evaluation Unlocks Any-Step Text-to-Image Generation
- Co-GRPO: Co-Optimized Group Relative Policy Optimization for Masked Diffusion Model
- Manifold-Constrained Noise Optimization for Diverse Diffusion Sampling
- Normalizing Trajectory Models
- UniGen-AR: Unifying Visual Generation with Auto-Regressive Modeling
- Argus-Unified: Towards A Compact and Economical Unified Model for Image Understanding and Generation
- Harm is not Universal: Community-Specific Toxicity Detection is Urgently Needed
- NormGuard: Reward-Preserving Norm Constraints in Flow-Matching Reinforcement Learning
- PixelGen: Improving Pixel Diffusion with Perceptual Supervision
- CRAFT: Continuous Reasoning and Agentic Feedback Tuning for Multimodal Text-to-Image Generation
- Visual-Aware CoT: Achieving High-Fidelity Visual Consistency in Unified Models
- VA-π: Variational Policy Alignment for Pixel-Aware Autoregressive Generation
- dMLLM-TTS: Self-Verified and Efficient Test-Time Scaling for Diffusion Multi-Modal Large Language Models
- VisionDirector: Vision-Language Guided Closed-Loop Refinement for Generative Image Synthesis
- MixFlow Training: Alleviating Exposure Bias with Slowed Interpolation Mixture
- Both Semantics and Reconstruction Matter: Making Representation Encoders Ready for Text-to-Image Generation and Editing
- UmniBench: Unified Understand and Generation Model Oriented Omni-dimensional Benchmark
- Alchemist: Unlocking Efficiency in Text-to-Image Model Training via Meta-Gradient Data Selection
- GenEval 2: Addressing Benchmark Drift in Text-to-Image Evaluation
- FasterVAR: Plug-and-Play Acceleration for Visual Autoregressive Models
- Expand and Prune: Maximizing Trajectory Diversity for Effective GRPO in Generative Models
- ViewMask-1-to-3: Multi-View Consistent Image Generation via Multimodal Discrete Diffusion Models
- OUSAC: Optimized Guidance Scheduling with Adaptive Caching for DiT Acceleration
- Image Diffusion Preview with Consistency Solver
- STAR: STacked AutoRegressive Scheme for Unified Multimodal Learning
- Exploring the Design Space of Transition Matching
- Anchoring Values in Temporal and Group Dimensions for Flow Matching Model Alignment
- EditMGT: Unleashing Potentials of Masked Generative Transformers in Image Editing
- Infinity and Beyond: Compositional Alignment in VAR and Diffusion T2I Models
- Exploring MLLM-Diffusion Information Transfer with MetaCanvas
- SVG-T2I: Scaling Up Text-to-Image Latent Diffusion Model Without Variational Autoencoder
- DraCo: Draft as CoT for Text-to-Image Preview and Rare Concept Generation
- AgentComp: From Agentic Reasoning to Compositional Mastery in Text-to-Image Models
- Fast-ARDiff: An Entropy-informed Acceleration Framework for Continuous Space Autoregressive Generation
- Position: Universal Aesthetic Alignment Narrows Artistic Expression
- Chain-of-Image Generation: Toward Monitorable and Controllable Image Generation
- SJD++: Improved Speculative Jacobi Decoding for Training-free Acceleration of Discrete Auto-regressive Text-to-Image Generation
- ParaUni: Enhance Generation in Unified Multimodal Model with Reinforcement-driven Hierarchical Parallel Information Interaction
- An Efficient Test-Time Scaling Approach for Image Generation
- Aligned but Stereotypical? The Hidden Influence of System Prompts on Social Bias in LVLM-Based Text-to-Image Models
- Autoregressive Image Generation Needs Only a Few Lines of Cached Tokens
- EMMA: Efficient Multimodal Understanding, Generation, and Editing with a Unified Architecture
- TwinFlow: Realizing One-step Generation on Large Models with Self-adversarial Flows
- Highly Efficient Test-Time Scaling for T2I Diffusion Models with Text Embedding Perturbation
- Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Models
- Glance: Accelerating Diffusion Models with 1 Sample
- WeMMU: Enhanced Bridging of Vision-Language Models and Diffusion Models via Noisy Query Tokens
- TUNA: Taming Unified Visual Representations for Native Unified Multimodal Models
- AlignVid: Training-Free Attention Scaling for Semantic Fidelity in Text-Guided Image-to-Video Generation
- DCText: Scheduled Attention Masking for Visual Text Generation via Divide-and-Conquer Strategy
- RealGen: Photorealistic Text-to-Image Generation via Detector-Guided Rewards
- InvarDiff: Cross-Scale Invariance Caching for Accelerated Diffusion Models
- FR-TTS: Test-Time Scaling for NTP-based Image Generation with Effective Filling-based Reward Signal
- Visual Generation Tuning
- VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstruction
- Ovis-Image Technical Report
- Test-time scaling of diffusions with flow maps
- AIA: Rethinking Architecture Decoupling Strategy In Unified Multimodal Model
- Designing Instance-Level Sampling Schedules via REINFORCE with James-Stein Shrinkage
- DiverseVAR: Balancing Diversity and Quality of Next-Scale Visual Autoregressive Models
- Progress by Pieces: Test-Time Scaling for Autoregressive Image Generation
- PixelDiT: Pixel Diffusion Transformers for Image Generation
- MapReduce LoRA: Advancing the Pareto Front in Multi-Preference Optimization for Generative Models
- Does Understanding Inform Generation in Unified Multimodal Models? From Analysis to Path Forward
- HBridge: H-Shape Bridging of Heterogeneous Experts for Unified Multimodal Understanding and Generation
- The Image as Its Own Reward: Reinforcement Learning with Adversarial Reward for Image Generation
- UniGame: Turning a Unified Multimodal Model Into Its Own Adversary
- DeCo: Frequency-Decoupled Pixel Diffusion for End-to-End Image Generation
- Synthetic Curriculum Reinforces Compositional Text-to-Image Generation
- ConsistCompose: Unified Multimodal Layout Control for Image Composition
- MammothModa2: A Unified AR-Diffusion Framework for Multimodal Understanding and Generation
- Diversity Has Always Been There in Your Visual Autoregressive Models
- Pluggable Pruning with Contiguous Layer Distillation for Diffusion Transformers
- AMS-KV: Adaptive KV Caching in Multi-Scale Visual Autoregressive Transformers
- Distribution Matching Distillation Meets Reinforcement Learning
- Mixture of States: Routing Token-Level Dynamics for Multimodal Generation
- MMaDA-Parallel: Multimodal Large Diffusion Language Models for Thinking-Aware Editing and Generation
- Perturb a Model, Not an Image: Towards Robust Privacy Protection via Anti-Personalized Diffusion Models
- oboro: Text-to-Image Synthesis on Limited Data using Flow-based Diffusion Transformer with MMH Attention
- Exploring the AI Obedience: Why is Generating a Pure Color Image Harder than CyberPunk?
- NextFlow: Unified Sequential Modeling Activates Multimodal Understanding and Generation
- Generating an Image From 1,000 Words: Enhancing Text-to-Image With Structured Captions
- InfinityStar: Unified Spacetime AutoRegressive Modeling for Visual Generation
- Diffusion-SDPO: Safeguarded Direct Preference Optimization for Diffusion Models
- ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generation
- E-MMDiT: Revisiting Multimodal Diffusion Transformer Design for Fast Image Synthesis under Limited Resources
- Back to the Communities: A Mixed-Methods and Community-Driven Evaluation of Cultural Sensitivity in Text-to-Image Models
- Emu3.5: Native Multimodal Models are World Learners
- PairUni: Pairwise Training for Unified Multimodal Language Models
- Anchoring and Steering Diffusion: Enhancing the Faithfulness of Text-to-Image Generation at Inference Time
- Amortized Moment Matching for Visual Generation
- High-Fidelity Two-Step Image Generation via Teacher-Aligned End-to-End Distillation
- AlphaGRPO: Unlocking Self-Reflective Multimodal Generation in UMMs via Decompositional Verifiable Reward
- Calibri: Enhancing Diffusion Transformers via Parameter-Efficient Calibration
- Uniform Discrete Diffusion with Metric Path for Video Generation
- Routing Matters in MoE: Scaling Diffusion Transformers with Explicit Routing Guidance
- Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generation
- UltraImageGen: Efficient Ultra-High-Resolution Image Generation with Hierarchical Local Attention
- Compositional Image Synthesis with Inference-Time Scaling
- M3T2IBench: A Large-Scale Multi-Category, Multi-Instance, Multi-Relation Text-to-Image Benchmark
- FairJudge: MLLM Judging for Social Attributes and Prompt Image Alignment
- GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clipping
- Sample By Step, Optimize By Chunk: Chunk-Level GRPO For Text-to-Image Generation
- Towards a Golden Classifier-Free Guidance Path via Foresight Fixed Point Iterations
- EditInfinity: Image Editing with Binary-Quantized Generative Models
- GenColorBench: A Color Evaluation Benchmark for Text-to-Image Generation Models
- Modal Aphasia: Can Unified Multimodal Models Describe Images From Memory?
- SSD: Spatial-Semantic Head Decoupling for Efficient Autoregressive Image Generation
- UniGenBench++: A Unified Semantic Evaluation Benchmark for Text-to-Image Generation
- UniRL-Zero: Reinforcement Learning on Unified Models with Joint Language Model and Diffusion Model Experts
- L2P: Unlocking Latent Potential for Pixel Generation
- Generation then Reconstruction: Accelerating Masked Autoregressive Models via Two-Stage Sampling
- Visual Autoregressive Models Beat Diffusion Models on Inference Time Scaling
- BLIP3o-NEXT: Next Frontier of Native Image Generation
- Constantly Improving Image Models Need Constantly Improving Benchmarks
- End-to-End Multi-Modal Diffusion Mamba
- Reinforcement Learning Meets Masked Generative Models: Mask-GRPO for Text-to-Image Generation
- NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matching
- UniFusion: Vision-Language Model as Unified Encoder in Image Generation
- SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Models
- Improving Text-to-Image Generation with Input-Side Inference-Time Scaling
- Demystifying Numerosity in Diffusion Models -- Limitations and Remedies
- GIR-Bench: Versatile Benchmark for Generating Images with Reasoning
- OmniQuality-R: Advancing Reward Models Through All-Encompassing Quality Assessment
- UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generation
- Head-wise Adaptive Rotary Positional Encoding for Fine-Grained Image Generation
- Asymmetric Flow Models
- Dense2MoE: Restructuring Diffusion Transformer to MoE for Efficient Text-to-Image Generation
- Speculative Jacobi-Denoising Decoding for Accelerating Autoregressive Text-to-image Generation
- UniVideo: Unified Understanding, Generation, and Editing for Videos
- Reinforcing Diffusion Models by Direct Group Preference Optimization
- Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistency
- GenPilot: A Multi-Agent System for Test-Time Prompt Optimization in Image Generation
- Growing Visual Generative Capacity for Pre-Trained MLLMs
- Ming-UniVision: Joint Image Understanding and Generation with a Unified Continuous Tokenizer
- VUGEN: Visual Understanding priors for GENeration
- Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understanding
- Improving Chain-of-Thought Efficiency for Autoregressive Image Generation
- Factuality Matters: When Image Generation and Editing Meet Structured Visuals
- ConceptSplit: Decoupled Multi-Concept Personalization of Diffusion Models via Token-wise Adaptation and Attention Disentanglement
- Asynchronous Denoising Diffusion Models for Aligning Text-to-Image Generation
- Diverse Text-to-Image Generation via Contrastive Noise Optimization
- Consolidating Reinforcement Learning for Multimodal Discrete Diffusion Models
- Fine-Tuning Diffusion Models via Intermediate Distribution Shaping
- Smart-GRPO: Smartly Sampling Noise for Efficient RL of Flow-Matching Models
- SoftCFG: Uncertainty-guided Stable Guidance for Visual Autoregressive Model
- Plug-and-Play Prompt Refinement via Latent Feedback for Diffusion Model Alignment
- MetaLogic: Robustness Evaluation of Text-to-Image Models via Logically Equivalent Prompts
- Stitch: Training-Free Position Control in Multimodal Diffusion Transformers
- Query-Kontext: An Unified Multimodal Model for Image Generation and Editing
- Go with Your Gut: Scaling Confidence for Autoregressive Image Generation
- IMG: Calibrating Diffusion Models via Implicit Multimodal Guidance
- IRIS: Intrinsic Reward Image Synthesis
- Aligning Visual Foundation Encoders to Tokenizers for Diffusion Models
- Score Distillation of Flow Matching Models
- STAGE: Stable and Generalizable GRPO for Autoregressive Image Generation
- RealUnify: Do Unified Models Truly Benefit from Unification? A Comprehensive Benchmark
- CLQ: Cross-Layer Guided Orthogonal-based Quantization for Diffusion Transformers
- Uni-X: Mitigating Modality Conflict with a Two-End-Separated Architecture for Unified Multimodal Models
- GLASS Flows: Transition Sampling for Alignment of Flow and Diffusion Models
- Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Models
- UniAlignment: Semantic Alignment for Unified Image Generation, Understanding, Manipulation and Perception
- HunyuanImage 3.0 Technical Report
- Soft-Di[M]O: Improving One-Step Discrete Image Generation with Soft Embeddings
- Group Critical-token Policy Optimization for Autoregressive Image Generation
- MILR: Improving Multimodal Image Generation via Test-Time Latent Reasoning
- RAPID3: Tri-Level Reinforced Acceleration Policies for Diffusion Transformer
- FailureAtlas:Mapping the Failure Landscape of T2I Models via Active Exploration
- Discrete Guidance Matching: Exact Guidance for Discrete Flow Matching
- SD3.5-Flash: Distribution-Guided Distillation of Generative Flows
- Hallucination as an Upper Bound: A New Perspective on Text-to-Image Evaluation
- EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning
- FeatFix: Reuse What You Verify through Local Exact-Feature Correction for Faster Cached Diffusion Inference
- Hyper-Bagel: A Unified Acceleration Framework for Multimodal Understanding and Generation
- FlexSED: Towards Open-Vocabulary Sound Event Detection
- OmniBridge: Unified Multimodal Understanding, Generation, and Retrieval via Latent Space Alignment
- MEF: A Systematic Evaluation Framework for Text-to-Image Models
- CARINOX: Inference-time Scaling with Category-Aware Reward-based Initial Noise Optimization and Exploration
- MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer
- AcT2I: Evaluating and Improving Action Depiction in Text-to-Image Models
- DiffusionNFT: Online Diffusion Reinforcement with Forward Process
- MaskAttn-SDXL: Controllable Region-Level Text-To-Image Generation
- T2I-ReasonBench: Benchmarking Reasoning-Informed Text-to-Image Generation
- UnifiedVisual: A Framework for Constructing Unified Vision-Language Datasets
- AToken: A Unified Tokenizer for Vision
- GenExam: A Multidisciplinary Text-to-Image Exam
- SpeCa: Accelerating Diffusion Transformers with Speculative Feature Caching
- MagicMirror: A Large-Scale Dataset and Benchmark for Fine-Grained Artifacts Assessment in Text-to-Image Generation
- FLUX-Reason-6M & PRISM-Bench: A Million-Scale Text-to-Image Reasoning Dataset and Comprehensive Benchmark
- Unified Multimodal Model as Auto-Encoder
- Reconstruction Alignment Improves Unified Multimodal Models
- Directly Aligning the Full Diffusion Trajectory with Fine-Grained Human Preference
- Interleaving Reasoning for Better Text-to-Image Generation
- Coefficients-Preserving Sampling for Reinforcement Learning with Flow Matching
- From Image Generation to Infrastructure Design: a Multi-agent Pipeline for Street Design Generation
- The Telephone Game: Evaluating Semantic Drift in Unified Models
- Transition Models: Rethinking the Generative Learning Objective
- Skywork UniPic 2.0: Building Kontext Model with Online RL for Unified Multimodal Model
- OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generation
- Easier Painting Than Thinking: Can Text-to-Image Models Set the Stage, but Not Direct the Play?
- Data-Driven Loss Functions for Inference-Time Optimization in Text-to-Image Generation
- Draw-In-Mind: Rebalancing Designer-Painter Roles in Unified Multimodal Models Benefits Image Editing
- InfoScale: Unleashing Training-free Variable-scaled Image Generation via Effective Utilization of Information
- Inference-Time Alignment Control for Diffusion Models with Reinforcement Learning Guidance
- Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learning
- Visual-CoG: Stage-Aware Reinforcement Learning with Chain of Guidance for Text-to-Image Generation
- Scaling Group Inference for Diverse and High-Quality Generation
- SAGA: Learning Signal-Aligned Distributions for Improved Text-to-Image Generation
- Navigating the Exploration-Exploitation Tradeoff in Inference-Time Scaling of Diffusion Models
- NextStep-1: Toward Autoregressive Image Generation with Continuous Tokens at Scale
- A Unified Multi-Agent Framework for Universal Multimodal Understanding and Generation
- Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generation
- Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Models
- Images Speak Louder Than Scores: Failure Mode Escape for Enhancing Generative Quality
- Reinforcement Learning for Large Model: A Survey
- TBAC-UniImage: Unified Understanding and Generation by Ladder-Side Diffusion Tuning
- AR-GRPO: Training Autoregressive Image Generation Models via Reinforcement Learning
- MultiRef: Controllable Image Generation with Multiple Visual References
- Multimodal learning with next-token prediction for large multimodal models
- Elastic Diffusion Transformer
- VISTAR:A User-Centric and Role-Driven Benchmark for Text-to-Image Evaluation
- Bifrost-1: Bridging Multimodal LLMs and Diffusion Models with Patch-level CLIP Latents
- Uni-cot: Towards Unified Chain-of-Thought Reasoning Across Text and Vision
- MetaDiT: Enabling Fine-grained Constraints in High-degree-of Freedom Metasurface Design
- HierarchicalPrune: Position-Aware Compression for Large-Scale Diffusion Models
- TempFlow-GRPO: When Timing Matters for GRPO in Flow Models
- Skywork UniPic: Unified Autoregressive Modeling for Visual Understanding and Generation
- Qwen-Image Technical Report
- Forecasting When to Forecast: Accelerating Diffusion Models with Confidence-Gated Taylor
- AttriCtrl: Fine-Grained Control of Aesthetic Attribute Intensity in Diffusion Models
- The Promise of RL for Autoregressive Image Editing
- PixNerd: Pixel Neural Field Diffusion
- UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editing
- X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Again
Related