IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs
2025/11/06 by Faraz, Ali, Saiqa Khan, Akash +13 · 2 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2511.04727
openalex publication_date 2025/11/06 · openalex created_date 2025/11/11 · openalex updated_date 2026/07/28
Abstract
Vision-language models (VLMs) have demonstrated impressive generalization across multimodal tasks, yet most evaluation benchmarks remain Western-centric, leaving open questions about their performance in culturally diverse and multilingual settings. To address this gap, we introduce IndicVisionBench, the first large-scale benchmark centered on the Indian subcontinent. Covering English and 10 Indian languages, our benchmark spans 3 multimodal tasks, including Optical Character Recognition (OCR), Multimodal Machine Translation (MMT), and Visual Question Answering (VQA), covering 6 kinds of question types. Our final benchmark consists of a total of ~5K images and 37K+ QA pairs across 13 culturally grounded topics. In addition, we release a paired parallel corpus of annotations across 10 Indic languages, creating a unique resource for analyzing cultural and linguistic biases in VLMs. We evaluate a broad spectrum of 8 models, from proprietary closed-source systems to open-weights medium and large-scale models. Our experiments reveal substantial performance gaps, underscoring the limitations of current VLMs in culturally diverse contexts. By centering cultural diversity and multilinguality, IndicVisionBench establishes a reproducible evaluation framework that paves the way for more inclusive multimodal research.
Citations
- How to Evaluate Automatic Speech Recognition: Comparing Different Performance and Bias Measures
- Behind Maya: Building a Multilingual Vision Language Model
- Gemma 3 Technical Report
- Chitrarth: Bridging Vision and Language for a Billion People
- SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
- OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning
- Maya: An Instruction Finetuned Multilingual Multimodal Model
- Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages
- Repetition Neurons: How Do Language Models Produce Repetitions?
- WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisines
- mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understanding
- xGen-MM (BLIP-3): A Family of Open Large Multimodal Models
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
- CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark
- Parrot: Multilingual Visual Instruction Tuning
- M4U: Evaluating Multilingual Understanding and Reasoning for Large Multimodal Models
- What matters when building vision-language models?
- How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites
- mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding
- EXAMS-V: A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Models
- DeepSeek-VL: Towards Real-World Vision-Language Understanding
- Peacock: A Family of Arabic Multimodal Large Language Models and Benchmarks
- PALO: A Polyglot Large Multimodal Model for 5B People
- Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs
- DocLLM: A layout-aware generative language model for multimodal document understanding
- Improved Baselines with Visual Instruction Tuning
- Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
- MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
- DocTr: Document Transformer for Structured Information Extraction in Documents
- MMBench: Is Your Multi-modal Model an All-around Player?
- MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
- M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models
- Visual Instruction Tuning
- Sigmoid Loss for Language Image Pre-Training
- Flamingo: a Visual Language Model for Few-Shot Learning
- FLAVA: A Foundational Language And Vision Alignment Model
- OCR-free Document Understanding Transformer
- Visually Grounded Reasoning across Languages and Cultures
- TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models
- Broaden the Vision: Geo-Diverse Visual Commonsense Reasoning
- xGQA: Cross-Lingual Visual Question Answering
- Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation
- Learning Transferable Visual Models From Natural Language Supervision
- DocVQA: A Dataset for VQA on Document Images
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- HuggingFace's Transformers: State-of-the-art Natural Language Processing
- Scene Text Visual Question Answering
- FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents
- The MeMAD Submission to the WMT18 Multimodal Translation Task
- Imagination improves Multimodal Translation
- Incorporating Global Visual Features into Attention-Based Neural Machine\n Translation
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
- Multi30K: Multilingual English-German Image Descriptions
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
- VQA: Visual Question Answering
- Evaluation of Deep Convolutional Nets for Document Image Classification and Retrieval
- Where are your Manners? Sharing Best Community Practices in the Web 2.0
Cited by
Related