SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension
2023/07/30 by Li, Bohao, Wang, Rui, Wang, Guangzhi +3 · 127 citations
#Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences
paper · doi:10.48550/arxiv.2307.16125
Abstract
Based on powerful Large Language Models (LLMs), recent generative Multimodal Large Language Models (MLLMs) have gained prominence as a pivotal research area, exhibiting remarkable capability for both comprehension and generation. In this work, we address the evaluation of generative comprehension in MLLMs as a preliminary step towards a comprehensive assessment of generative models, by introducing a benchmark named SEED-Bench. SEED-Bench consists of 19K multiple choice questions with accurate human annotations (x 6 larger than existing benchmarks), which spans 12 evaluation dimensions including the comprehension of both the image and video modality. We develop an advanced pipeline for generating multiple-choice questions that target specific evaluation dimensions, integrating both automatic filtering and manual verification processes. Multiple-choice questions with groundtruth options derived from human annotation enables an objective and efficient assessment of model performance, eliminating the need for human or GPT intervention during evaluation. We further evaluate the performance of 18 models across all 12 dimensions, covering both the spatial and temporal understanding. By revealing the limitations of existing MLLMs through evaluation results, we aim for SEED-Bench to provide insights for motivating future research. We will launch and consistently maintain a leaderboard to provide a platform for the community to assess and investigate model capability.
Cited by
- Same or Not? Enhancing Visual Perception in Vision-Language Models
- Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion Language Model Backbone
- Masking Teacher and Reinforcing Student for Distilling Vision-Language Models
- Unbiased Visual Reasoning with Controlled Visual Inputs
- Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model
- MarineEval: Assessing the Marine Intelligence of Vision-Language Models
- Benchmarking and Enhancing VLM for Compressed Image Understanding
- Adaptive-VoCo: Complexity-Aware Visual Token Compression for Vision-Language Models
- Differences That Matter: Auditing Models for Capability Gap Discovery and Rectification
- Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Models
- The Perceptual Observatory Characterizing Robustness and Grounding in MLLMs
- DiffusionVL: Translating Any Autoregressive Models into Diffusion Vision Language Models
- JMMMU-Pro: Image-based Japanese Multi-discipline Multimodal Understanding Benchmark via Vibe Benchmark Construction
- SDAR-VL: Stable and Efficient Block-wise Diffusion for Vision-Language Understanding
- HyperVL: An Efficient and Dynamic Multimodal Large Language Model for Edge Devices
- Video Reality Test: Can AI-Generated ASMR Videos fool VLMs and Humans?
- STAR: STacked AutoRegressive Scheme for Unified Multimodal Learning
- JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation
- FysicsWorld: A Unified Full-Modality Benchmark for Any-to-Any Understanding, Generation, and Reasoning
- AgriGPT-Omni: A Unified Speech-Vision-Text Framework for Multilingual Agricultural Intelligence
- Towards Lossless Ultimate Vision Token Compression for VLMs
- HybridToken-VLM: Hybrid Token Compression for Vision-Language Models
- Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models
- GeoViS: Geospatially Rewarded Visual Search for Remote Sensing Visual Grounding
- VLM-Pruner: Buffering for Spatial Sparsity in an Efficient VLM Centrifugal Token Pruning Paradigm
- TUNA: Taming Unified Visual Representations for Native Unified Multimodal Models
- See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Models
- VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstruction
- From Pixels to Feelings: Aligning MLLMs with Human Cognitive Perception of Images
- CLASH: A Benchmark for Cross-Modal Contradiction Detection
- VITAL: Vision-Encoder-centered Pre-training for LMMs in Visual Quality Assessment
- Multimodal Evaluation of Russian-language Architectures
- O3SLM: Open Weight, Open Data, and Open Vocabulary Sketch-Language Model
- Multimodal Large Language Models as Image Classifiers
- CoLM: Collaborative Large Models via A Client-Server Paradigm
- DeepEyesV2: Toward Agentic Multimodal Model
- MM-OPERA: Benchmarking Open-ended Association Reasoning for Large Vision-Language Models
- Do Vision-Language Models Measure Up? Benchmarking Visual Measurement Reading with MeasureBench
- VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMs
- SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs
- SSL4RL: Revisiting Self-supervised Learning as Intrinsic Reward for Visual-Language Reasoning
- MergeMix: A Unified Augmentation Paradigm for Visual and Multi-Modal Understanding
- Positional Preservation Embedding for Multimodal Large Language Models
- Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigation
- KBE-DME: Dynamic Multimodal Evaluation via Knowledge Enhanced Benchmark Evolution
- NoisyGRPO: Incentivizing Multimodal CoT Reasoning via Noise Injection and Bayesian Estimation
- GranViT: A Fine-Grained Vision Model With Autoregressive Perception For MLLMs
- HyperET: Efficient Training in Hyperbolic Space for Multi-modal Large Language Models
- Unified Reinforcement and Imitation Learning for Vision-Language Models
- ProCLIP: Progressive Vision-Language Alignment via LLM-based Embedder
- UNO-Bench: A Unified Benchmark for Exploring the Compositional Law Between Uni-modal and Omni-modal in Omni Models
- UWBench: A Comprehensive Vision-Language Benchmark for Underwater Understanding
- UniRL-Zero: Reinforcement Learning on Unified Models with Joint Language Model and Diffusion Model Experts
- Token-Level Inference-Time Alignment for Vision-Language Models
- Res-Bench: Benchmarking the Robustness of Multimodal Large Language Models to Dynamic Resolution Input
- MultiVerse: A Multi-Turn Conversation Benchmark for Evaluating Large Vision and Language Models
- From Pixels to Words -- Towards Native Vision-Language Primitives at Scale
- Vision-Centric Activation and Coordination for Multimodal Large Language Models
- VQArt-Bench: A semantically rich VQA Benchmark for Art and Cultural Heritage
- FlexAC: Towards Flexible Control of Associative Reasoning in Multimodal Large Language Models
- RAG-IGBench: Innovative Evaluation for RAG-based Interleaved Generation in Open-domain Question Answering
- MomentSeg: Moment-Centric Sampling for Enhanced Video Pixel Understanding
- How to Teach Large Multimodal Models New Skills
- Growing Visual Generative Capacity for Pre-Trained MLLMs
- TTRV: Test-Time Reinforcement Learning for Vision Language Models
- Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understanding
- Human-MME: A Holistic Evaluation Benchmark for Human-Centric Multimodal Large Language Models
- Importance Sampling for Multi-Negative Multimodal Direct Preference Optimization
- Training-Free Token Pruning via Zeroth-Order Gradient Estimation in Vision-Language Models
- Uni-X: Mitigating Modality Conflict with a Two-End-Separated Architecture for Unified Multimodal Models
- When MLLMs Meet Compression Distortion: A Coding Paradigm Tailored to MLLMs
- VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenes
- HiViS: Hiding Visual Tokens from the Drafter for Speculative Decoding in Vision-Language Models
- Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning
- RIV: Recursive Introspection Mask Diffusion Vision Language Model
- Multi-modal Data Spectrum: Multi-modal Datasets are Multi-dimensional
- Uncovering Intrinsic Capabilities: A Paradigm for Data Curation in Vision-Language Models
- A Structured Framework for Evaluating and Enhancing Interpretive Capabilities of Multimodal LLMs in Culturally Situated Tasks
- MMPB: It's Time for Multi-Modal Personalization
- Multilingual Vision-Language Models, A Survey
- Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach
- Discrete Guidance Matching: Exact Guidance for Discrete Flow Matching
- Inference-Time Agentic Decision Rules Beat Longer Evolving Search for Multi-Image Medical Reasoning
- The Artificial Intelligence Cognitive Examination: A Survey on the Evolution of Multimodal Evaluation From Recognition to Reasoning
- Unveiling Chain of Step Reasoning for Vision-Language Models with Fine-grained Rewards
- MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer
- Qianfan-VL: Domain-Enhanced Universal Vision-Language Models
- A Multi-To-One Interview Paradigm for Efficient MLLM Evaluation
- Towards Rationale-Answer Alignment of LVLMs via Self-Rationale Calibration
- AsyMoE: Leveraging Modal Asymmetry for Enhanced Expert Specialization in Large Vision-Language Models
- 3D Aware Region Prompted Vision Language Model
- Seeing is Not Understanding: A Benchmark on Perception-Cognition Disparities in Large Language Models
- VARCO-VISION-2.0 Technical Report
- Towards Understanding Visual Grounding in Visual Language Models
- Reconstruction Alignment Improves Unified Multimodal Models
- Towards Open World Detection: A Survey
- Promptception: How Sensitive Are Large Multimodal Models to Prompts?
- OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generation
- OpenVision 2: A Family of Generative Pretrained Visual Encoders for Multimodal Learning
- Safe-LLaVA: A Privacy-Preserving Vision-Language Dataset and Benchmark for Biometric Safety
- Waste-Bench: A Comprehensive Benchmark for Evaluating VLLMs in Cluttered Environments
- BLUEX Revisited: Enhancing Benchmark Coverage with Automatic Captioning
- Mitigating Hallucinations in Multimodal LLMs via Object-aware Preference Optimization
- MMTok: Multimodal Coverage Maximization for Efficient Inference of VLMs
- Unveiling Trust in Multimodal Large Language Models: Evaluation, Analysis, and Mitigation
- Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language-Vision Models
- HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes
- MoIIE: Mixture of Intra- and Inter-Modality Experts for Large Vision Language Models
- SHALE: A Scalable Benchmark for Fine-grained Hallucination Evaluation in LVLMs
- MCITlib: Multimodal Continual Instruction Tuning Library and Benchmark
- Δ-AttnMask: Attention-Guided Masked Hidden States for Efficient Data Selection and Augmentation
- Bifrost-1: Bridging Multimodal LLMs and Diffusion Models with Patch-level CLIP Latents
- SIFThinker: Spatially-Aware Image Focus for Visual Reasoning
- VFlowOpt: A Token Pruning Framework for LMMs with Visual Information Flow-Guided Optimization
- Boosting Visual Knowledge-Intensive Training for LVLMs Through Causality-Driven Visual Object Completion
- OmniPlay: Benchmarking Omni-Modal Models on Omni-Modal Game Playing
- Evaluating Variance in Visual Question Answering Benchmarks
- A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Models
- Mitigating Information Loss under High Pruning Rates for Efficient Large Vision Language Models
- Instruction-Grounded Visual Projectors for Continual Learning of Generative Vision-Language Models
- Bidirectional Likelihood Estimation with Multi-Modal Large Language Models for Text-Video Retrieval
- HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Models
- MAGE: Multimodal Alignment and Generation Enhancement via Bridging Visual and Semantic Spaces
- TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction in MLLMs
- ARC-Hunyuan-Video-7B: Structured Video Comprehension of Real-World Shorts
- TransPrune: Token Transition Pruning for Efficient Large Vision-Language Model
- A Survey of Token Compression for Efficient Multimodal Large Language Models
Related