PM4Bench: A Parallel Multilingual Multi-Modal Multi-task Benchmark for Large Vision Language Model

2025/03/24 by Gao, Junyuan, Song, Jiahe, Wu, Jiang +11 · 1 citation
#Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2503.18484

Abstract

Existing multilingual benchmarks for Large Vision Language Models (LVLMs) suffer from limitations including language-specific content biases, disjointed multimodal input formats, and a lack of safety evaluation. To address these gaps, we propose PM4Bench, the first Parallel Multilingual Multi-Modal Multi-task Benchmark for LVLMs. PM4Bench features a parallel corpus design across 10 languages, enabling fair and accurate cross-lingual comparisons. It includes the vision setting where text and queries are embedded in images, requiring LVLMs to simultaneously "see", "read", and "think", aligning with real-world applications. Additionally, PM\textsuperscript4Bench incorporates safety evaluations, addressing critical oversight in existing multilingual benchmarks. Using PM4Bench, we evaluate 11 mainstream LVLMs, revealing significant cross-linguistic performance disparities, particularly in vision settings, and identifying OCR capability as a key determinant of these imbalances. We will release PM4Bench at https://github.com/opendatalab/PM4Bench .

Cited by

• RxnCaption: Reformulating Reaction Diagram Parsing as Visual Prompt Guided Captioning

Related

• MVL-SIB: A Massively Multilingual Vision-Language Benchmark for Cross-Modal Topical Matching
• MMBench: Is Your Multi-modal Model an All-around Player?
• Scalable Performance Analysis for Vision-Language Models
• Centurio: On Drivers of Multilingual Ability of Large Vision-Language Model
• How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites