Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Following
2025/11/26 by Xiong, Tianyi, Ge, Yi, Li, Ming +13 · 2 citations
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences
paper · doi:10.48550/arxiv.2511.21662
Abstract
Large multimodal models (LMMs) are increasingly adopted as judges in multimodal evaluation systems due to their strong instruction following and consistency with human preferences. However, their ability to follow diverse, fine-grained evaluation criteria remains underexplored. We develop Multi-Crit, a benchmark for evaluating multimodal judges on their capacity to follow pluralistic criteria and produce reliable criterion-level judgments. Covering both open-ended generation and verifiable reasoning tasks, Multi-Crit is built through a rigorous data curation pipeline that gathers challenging response pairs with multi-criterion human annotations. It further introduces three novel metrics for systematically assessing pluralistic adherence, criterion-switching flexibility, and the ability to recognize criterion-level preference conflicts. Comprehensive analysis of 25 LMMs reveals that 1) proprietary models still struggle to maintain consistent adherence to pluralistic criteria--especially in open-ended evaluation; 2) open-source models lag further behind in flexibly following diverse criteria; and 3) critic fine-tuning with holistic judgment signals enhances visual grounding but fails to generalize to pluralistic criterion-level judgment. Additional analyses on reasoning fine-tuning, test-time scaling, and boundary consistency between open-source and proprietary models further probe the limits of current multimodal judges. As a pioneering study, Multi-Crit lays the foundation for building reliable and steerable multimodal AI evaluation.
Citations
- GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation
- Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Image
- MM-CRITIC: A Holistic Evaluation of Large Multimodal Models as Multimodal Critique
- LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training
- RLBFF: Binary Flexible Feedback to bridge between Human Feedback & Verifiable Rewards
- BaseReward: A Strong Baseline for Multimodal Reward Model
- MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe
- Parallel-R1: Towards Parallel Thinking via Reinforcement Learning
- LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Model
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
- Cost-Aware Contrastive Routing for LLMs
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning
- ARGUS: Hallucination and Omission Evaluation in Video-LLMs
- Learning to Reason via Mixture-of-Thought for Logical Reasoning
- Qwen3 Technical Report
- Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning
- R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learning
- Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Models
- InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models
- VisualPuzzles: Decoupling Multimodal Reasoning Evaluation from Domain Knowledge
- SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models
- SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvement
- Kimi-VL Technical Report
- Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning
- Unified Reward Model for Multimodal Understanding and Generation
- Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Models
- Qwen2.5-VL Technical Report
- MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
- MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
- Improving Video Generation with Human Feedback
- InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model
- Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark
- Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives
- VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models
- GPT-4o System Card
- JudgeBench: A Benchmark for Evaluating LLM-based Judges
- RMB: Comprehensively Benchmarking Reward Models in LLM Alignment
- VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignment
- LLaVA-Critic: Learning to Evaluate Multimodal Models
- Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
- LLaVA-OneVision: Easy Visual Task Transfer
- The Llama 3 Herd of Models
- Improving Context-Aware Preference Modeling for Language Models
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
- WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences
- RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthiness
- Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement
- ImageInWords: Unlocking Hyper-Detailed Image Descriptions
- DOCCI: Descriptions of Connected and Contrasting Images
- MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?
- RewardBench: Evaluating Reward Models for Language Modeling
- MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark
- Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation
- InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
- GPT-4V(ision) as a Generalist Evaluator for Vision-Language Tasks
- GPT-4 Vision on Medical Image Classification -- A Case Study on COVID-19 Dataset
- HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models
- Aligning Large Multimodal Models with Factually Augmented RLHF
- MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
- Visual Instruction Tuning
- GPT-4 Technical Report
- MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
- Modeling Context in Referring Expressions
- Inter-Coder Agreement for Computational Linguistics
- OpenAI o1 System Card
Cited by
Related