FlagEvalMM: A Flexible Framework for Comprehensive Multimodal Model Evaluation

2025/06/10 by Zhiqiang He, He, Zheqi, Yesheng Liu +12 · 2 citations
Computer Science · #Multimodal Machine Learning Applications #Topic Modeling #Domain Adaptation and Few-Shot Learning

paper · pdf · doi:10.48550/arxiv.2506.09081

Abstract

We present FlagEvalMM, an open-source evaluation framework designed to comprehensively assess multimodal models across a diverse range of vision-language understanding and generation tasks, such as visual question answering, text-to-image/video generation, and image-text retrieval. We decouple model inference from evaluation through an independent evaluation service, thus enabling flexible resource allocation and seamless integration of new tasks and models. Moreover, FlagEvalMM utilizes advanced inference acceleration tools (e.g., vLLM, SGLang) and asynchronous data loading to significantly enhance evaluation efficiency. Extensive experiments show that FlagEvalMM offers accurate and efficient insights into model strengths and limitations, making it a valuable tool for advancing multimodal research. The framework is publicly accessible at https://github.com/flageval-baai/FlagEvalMM.

Citations

• Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling
• A Survey on LLM-as-a-Judge
• MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark
• MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilities
• VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models
• VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generation
• Evaluating Text-to-Visual Generation with Image-to-Text Generation
• CogView3: Finer and Faster Text-to-Image Generation via Relay Diffusion
• Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
• Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
• CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning
• VBench: Comprehensive Benchmark Suite for Video Generative Models
• MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Language Models are Few-Shot Learners
• Microsoft COCO: Common Objects in Context

Cited by

• Do Vision-Language Models Measure Up? Benchmarking Visual Measurement Reading with MeasureBench
• Beyond Benchmark: LLMs Evaluation with an Anthropomorphic and Value-oriented Roadmap

Related

• LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models
• MMBench: Is Your Multi-modal Model an All-around Player?
• Veagle: Advancements in Multimodal Representation Learning
• Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping
• MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilities