Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation

2024/01/12 by Seongyun Lee, Lee, Seongyun, Seungone Kim +7 · 25 citations
Computer Science · #Computation and Language (cs.CL) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2401.06591

openalex publication_date 2024/01/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Assessing long-form responses generated by Vision-Language Models (VLMs) is challenging. It not only requires checking whether the VLM follows the given instruction but also verifying whether the text output is properly grounded on the given image. Inspired by the recent approach of evaluating LMs with LMs, in this work, we propose to evaluate VLMs with VLMs. For this purpose, we present a new feedback dataset called the Perception Collection, encompassing 15K customized score rubrics that users might care about during assessment. Using the Perception Collection, we train Prometheus-Vision, the first open-source VLM evaluator model that can understand the user-defined score criteria during evaluation. Prometheus-Vision shows the highest Pearson correlation with human evaluators and GPT-4V among open-source models, showing its effectiveness for transparent and accessible evaluation of VLMs. We open-source our code, dataset, and model at https://github.com/kaistAI/prometheus-vision

Cited by

• LiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy Distillation
• A Reconstruction-Based Framework for Caption Evaluation Beyond Reference Captions
• DOCR-Inspector: Fine-Grained and Automated Evaluation of Document Parsing with VLM
• Becoming Experienced Judges: Selective Test-Time Learning for Evaluators
• M4-RAG: A Massive-Scale Multilingual Multi-Cultural Multimodal RAG
• PPTArena: A Benchmark for PowerPoint Editing
• A Progressive Evaluation Framework for Multicultural Analysis of Story Visualization
• Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Following
• Orion: A Unified Visual Agent for Multimodal Perception, Advanced Visual Reasoning and Execution
• LLM-as-a-Verifier: A General-Purpose Verification Framework
• Human-Agent Collaborative Paper-to-Page Crafting
• Disentanglement Beyond Static vs. Dynamic: A Benchmark and Evaluation Framework for Multi-Factor Sequential Representations
• MultiVerse: A Multi-Turn Conversation Benchmark for Evaluating Large Vision and Language Models
• Generative Universal Verifier as Multimodal Meta-Reasoner
• Active Model Selection for Large Language Models
• Deploying Tiny LVLM Judges for Real-World Evaluation of Chart Models: Lessons Learned and Best Practices
• From Filters to VLMs: Benchmarking Defogging Methods through Object Detection and Segmentation Performance
• Unlocking the Essence of Beauty: Advanced Aesthetic Reasoning with Relative-Absolute Policy Optimization
• VideoJudge: Bootstrapping Enables Scalable Supervision of MLLM-as-a-Judge for Video Understanding
• Instant Preference Alignment for Text-to-Image Diffusion Models
• Improving Large Vision and Language Models by Learning from a Panel of Peers
• PosterForest: Hierarchical Multi-Agent Collaboration for Scientific Poster Generation
• The Mind's Eye: A Multi-Faceted Reward Framework for Guiding Visual Metaphor Generation
• Personalized Safety Alignment for Text-to-Image Diffusion Models
• Learning an Efficient Multi-Turn Dialogue Evaluator from Multiple Judges

Related

• ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
• VHELM: A Holistic Evaluation of Vision Language Models
• LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models
• CHIRP: A Fine-Grained Benchmark for Open-Ended Response Evaluation in Vision-Language Models
• AutoBench-V: Can Large Vision-Language Models Benchmark Themselves?